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About This Guide 



I n January 2000, the National Institute for Literacy (NIFL) 

published Equipped for the Future Content Standards: What Adults 
Need to Know and be Able to Do in the 21st Century. Since then, 
people in states and adult education programs across the country 
who are using the 16 Equipped for the Future (EFF) Standards have 
been looking forward to the development of assessments that we can 
use to measure adults’ performance on the Standards. This guide, which 
highlights the EFF approach to assessment and illustrates this approach 
with assessments and tools for the EFF Read With Understanding 
Standard, will enable people to do just that. 

niff’s goal in developing the Standards — the culmination of a long- 
term, collaborative research initiative — was to help improve the quahty 
and results of the U.S. adult learning system through standards-based 
improvement. Policymakers and administrators knew that they would 
not be able to reap the full benefits of the Standards until they had 
assessments and an accountability framework that were aligned with 
the standards. Only then would there be the tools to fully implement 
this national consensus on what adults need to know and be able to do 
to be equipped for daily life in the 21st century. 



Developing the Assessment Framework 

For the last three years, the Equipped for the Future Assessment 
Consortium, led by SRI International and the Center for Fiteracy Studies 
at the University of Tennessee, Knoxville, has been coordinating an effort 
across the states to develop an assessment framework that would fit with 
our rigorous Standards for Standards and Guiding Principles for an 
Assessment Framework, developed by our partners in 1999. 



The 16 EFF Standards 



The EFF Standards include four 
categories of skills that adults 
need to draw from to carry out key 
activities at home, at work and in 
the community: 

Communication Skills 

• Read With Understanding 

• Convey Ideas in Writing 

• Speak So Others Can Understand 

• Listen Actively 

• Observe Critically 

Decision-Making Skills 

• Solve Problems and Make 
Decisions 

• Plan 

• Use Math to Solve Problems and 
Communicate 

Interpersonal Skills 

• Cooperate With Others 

• Guide Others 

• Advocate and Influence 

• Resolve Conflict and Negotiate 

Lifelong Learning Skills 

• Take Responsibility for Learning 

• Learn Through Research 

• Reflect and Evaluate 

• Use Information and 
Communications Technology 



We have worked systematically with development partners, including 
practitioners and their students, researchers, and assessment experts, to build a model 
for adult cognition and learning that would serve as foundation for assessments that 
are aligned with EFF Standards — assessments that allow teachers to measure and 
report on how well adults can use the integrated skill processes that make up the EFF 
Standards to reach goals and satisfy purposes in their lives. 



In addition, we have defined the EFF Continuum of Performance and begun 
developing levels of performance for each of the 16 Content Standards. We began 
by focusing on the five EFF Standards that are central to the U.S. Department of 
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Education’s National Reporting System for the Adult Education and Eamily Literacy 
Act (Title 11 of the Workforce Investment Act), which will be of highest interest 
to states: 

• Read With Understanding 

• Use Math to Solve Problems and Communicate 

• Listen Actively 

• Speak So Others Can Understand 

• Convey Ideas in Writing 

Last year, acting on the guidance of a core group of our technical and policy advisors, 
we focused on developing a working model of accountability assessment for one 
ELL Standard — Read With Understanding. This model, called the EFF Assessment 
Prototype, includes sample tools that practitioners can use to assess adult 
achievements and mastery in Read With Understanding to meet accountabihty 
requirements for adult education programs. While programs will be able to use 
these tools now for the Read With Understanding Standard, our larger, long-term 
purpose in developing the prototype is twofold: 

1. Creating a Model for Assessment Developers 

The ELL Assessment Prototype is a standardized assessment system that assessment 
developers can use as a model to produce assessments for all 16 ELL Standards for 
accountability purposes — and can adapt to develop assessments for other purposes. 
The assessments will be aligned with specific ELL Standards. This prototype includes: 

• A model of adult performance on the ELL Standards (the ELL Continuum 
of Performance) 

• Performance-level descriptors for each ELL Standard 

• Assessment design specifications for each level of performance 

• Model assessment tasks and scoring rubrics for each level of performance 

2. Guiding Poliqmiakers and Practitioners 

The ELL Assessment Prototype guides policymakers and practitioners in thinking 
about how standards and assessments can be used most effectively to support adult 
learning, teacher professional development and program improvement to produce 
higher levels of adult learning and achievement. This guidance includes: 

• A model of standards-based educational improvement 

• A discussion of our approach to developing assessments of performance on the 
ELL Standards 

• Materials to support training and professional development in using ELL Standards 
and assessments for the purposes of accountability and educational improvement 
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About the EFF Assessment Consortium 

The EFF Assessment Consortium is a partnership between SRI International (formerly 
the Stanford Research Institute, now an independent, not-for-profit research and 
development organization with headquarters in Menlo Park, Calif) and the Center 
for Fiteracy Studies at the University of Tennessee, KnoxviUe. 

NIFF funded the consortium to build on work toward defining an assessment 
framework for EFF Standards carried out by the Institute for Educational Feadership 
in 1999-2000. The consortium’s work also builds on and expands EFF field research to 
refine the EFF Standards and to define a continuum of performance for the Standards 
that has been coordinated by the Center for Fiteracy Studies since 1998. This includes 
the EFF/National Reporting System Joint Data Collection Project, supported in 
2000-02 by NIFF and the U.S. Department of Education’s Division of Adult Education 
and Fiteracy in the Office of Vocational and Adult Education. 

This project has been an important source of data for the consortium’s work on 
developing and validating an assessment framework for the 16 EFF Content Standards. 
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Guiding Principles for the Assessment Framework 



1. The EFF Assessment Framework must address 
multiple purposes for assessment. The framework 
must provide for: 

• Information on learner achievements and mastery 
that is useful to the learner as well as the teacher 
throughout the instructional process. 

• Information about what adults can do that is 
credible to employers, educational institutions 
and policymakers, as well as to adult learners 
themselves. 

• Information that is useful for program and system 
improvement and accountability. 

2. To address these multiple purposes, the 
EFF Assessment Framework must support a 
multidimensional, flexible and systemic approach 
to assessment. Teachers and programs will be able 
to choose from a range of tools — to be identified or 
developed — that enable them to accurately measure 
performance against EFF Standards and that are 
linked to one another, so that multiple assessments 
can provide a rich portrait of competence. 

3. The EFF Assessment Framework must address 
learning over a lifetime. Strategies for assessment 
and credentialing must take into account the fact that 
adults build skills over time (rather than all at once), 

in response to changes in their life situations. 
Certificates and other credentials must be modular, 
designed to define competence or mastery at a 
particular point, and within a framework that assumes 
continuing development of competence as skills, 
knowledge and understanding are further developed 
over time. 

4. Since EFF standards define skills all adults 
need in order to carry out their roles as workers 
and members of families and communities, the 
EFF Assessment Framework must address a 
single continuum of performance for all adults — 
including those with only minimal formal education 
and those with many years of formal education, 
including advanced degrees. 



5. Each level defined in the EFF Assessment 
Framework must communicate clearly what an 
adult at that level can do. Numerical levels don’t 
communicate meaning to external audiences. Grade 
levels seem to communicate a common picture of 
performance, but in fact the meaning behind the 
labels vary widely from community to community and 
state to state. Grade levels are particularly misleading 
when applied to adult performance, since they focus 
on developmental skill levels that don’t match the 
ways in which adults, with their broader background 
and range of experience, can combine skills and 
knowledge to perform effectively in daily life. 

6. The levels defined in the EFF Assessment 
Framework must be explicitly linked to key 
external measures of competence (such as 
certificates of mastery, NAALVIAL survey levels, 
diplomas and other credentials), and key pathways 
(e.g., entry to higher education and to employment 
as defined by occupational skill standards) so that 
adults and systems can rely on them as accurate 
predictors of real-world performance. 

7. The levels defined in the EFF Assessment 
Framework must be the products of a national 
consensus-building process that assures 
portability of certificates and credentials. 

8. Work on the development of this framework 
must maintain the strong customer focus that has 
distinguished the EFF Standards development 
process to date. It must be based on a broad, 
inclusive definition of maximizing accountability for 
all activities to all customers — starting with the 
adult learner. 
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How To Use This Guide 



whether you are a policymaker, a program manager, a teacher and/or an assessment 
developer, you will find this guide useful for your work in adult education. 



If you want to understand the theoretical and research underpinnings of the EFF 
Assessment Approach — and get a broad overview of the work in this guide — turn to: 

Section 1 : The EFF Approach to Assessment 

This section highlights the importance of aligning standards, assessments and 
accountabihty and describes our approach to developing assessments that are aligned 
with the EFF Standards. 

This section also introduces a theory of action for standards-based educational improvement 
developed by the National Research Council (NRC) and explains how this theory of action 
has guided our development of accountabihty assessments for the EFF Standards. 



If you want to know the details about the EFF Assessment Prototype — including its 
relationship to the U.S. Department of Education’s National Reporting System — turn to: 

Section 2: The Accountability Assessment Prototype for Read With 
Understanding 

This section presents a comprehensive illustration of the EFF approach to assessment and 
standards-based educational improvement — our Assessment Prototype for Read With 
Understanding. 

This section includes: 

• Performance-level descriptions for Read With Understanding 

• A chart showing correspondences between performance levels for EFF Read With 
Understanding and Educational Functioning Fevels for the U.S. Department of Education’s 
National Reporting System 

• Design specifications for assessments of each of the EFF Read With Understanding 
performance levels, including a model assessment task, scoring rubric and scored samples 
of performance for each level 
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If you want to know how to use the EFF Assessment Prototype to develop 
assessments and report results, turn to: 

Section 3: Guide To Using the EFF Assessment Prototype for 
Accountability Purposes 

This section contains guidance for selecting, administering, scoring, interpreting and 
reporting results on EFF accountability assessments. This section also presents a set of 
three scenarios for how the assessments in the EFF Assessment Prototype might be 
used in a state agency, adult education program and classroom for reporting 
educational gains to the National Reporting System. 



If you want to find out how to train teachers and administrators to use the EFF 
accountability assessments appropriately to ensure fair, valid and reliable results, 
turn to: 

Section 4: Guidelines and Materials for Training 

This section includes materials and guidelines for training teachers and others to 
administer, score and report scores on EFF accountability assessments. 



If you want to learn how teachers and other literacy professionals can use the 
EFF Assessment Prototype to create assessments that improve instruction, 
turn to: 

Section 5: Guide To Using EFF Assessments To Improve 
Teaching and Learning 

This section provides information to guide teachers and others in designing and using 
assessments for instructional purposes to prepare adults for accountability assessments. 
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SECTION 1 

The EFF Approach to Assessment 



A dult learning theory and practical experience tell us 

that adult learning is goal-directed. Adults come to adult 
education programs to develop the skills they need to 
accomplish particular purposes, such as getting a better job. 

Improving performance on standardized tests of reading or math or 
other basic skills is not the most important goal for most adult learners. 

Adult education programs strive to provide adult learners with the 
opportunities they need to improve their knowledge, skills and abilities 
in ways that will help them to accompHsh their life goals. Accountability 
for adult education programs can and should recognize the reahties of 
adults’ goals. Accountabihty can and should inform efforts to improve 
the effectiveness of adult education programs in helping adults meet 
their goals. 

We developed the EFF Standards and the EFF approach to assessment 
for accountability and educational improvement to serve these ends. 

Each EFF Standard is an applied learning standard that defines an 
integrated skill process that adults use to achieve goals or purposes in 
their lives. Applied learning standards require an approach to assessment 
that permits collection and evaluation of evidence that adults can 
purposefully apply their knowledge, skills and abilities to accompHsh meaningful goals. 

The first three Guiding Principles for the Assessment Framework (see page 4) call 
for developing a variety of linked, accurate measures of performance that can provide 
information that is useful for a variety of purposes and that can be used to mark points 
of achievement on a continuum of adult learning and development that extends over a 
lifetime. The most basic requirement for developing an assessment framework that 
fives up to these principles is aligning assessments to a common, well-articulated 
content framework — in this case the EFF Standards. 

The assessments in the EFF Assessment Prototype illustrate how the EFF Assessment 
Framework can be applied to designing assessments for the Read With Understanding 
Standard for program accountability and improvement. To put the components of 
the prototype in perspective, it’s important to understand how aligned standards, 
assessments and accountability work in supporting standards-based improvement. 



In this section you will learn 

about: 

■ How the EFF Continuum 

of Performance describes key 
dimensions of adult learning 

■ How levels of performance 
reflect adults’ increasing 
proficiency of each EFF 
Standard 

■ How the EFF performance 
descriptions facilitate 
assessment design 

■ Why performance 
assessments are appropriate 
for accountability and 
educational improvement 
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How Standards-Based Improvement Works 

The basic theory of standards-based improvement — that content standards, 
aligned assessments and accountability mechanisms will provide clear expectations 
for learning and the motivation that will lead to continually higher levels of learning 
and achievement — is a relatively recent and profound shift in educational thinking. 
Although the history of standards-based improvement is relatively short in K-12 
schools and even shorter in adult basic education, there is an emerging body of 
research that reveals many important lessons about how and under what conditions 
standards-based improvement can work to produce higher levels of learning. 

In 1997, the NRC Board on Testing and Assessment created a Committee on Title 1 
Testing and Assessment to review available research and summarize findings on how 
the theory of standards-based improvement had played out in practice. In its report. 
Testing, Teaching, and Learning: A Guide for States and School Districts (NRC, 1999a), 
the committee paid particular attention to identifying the conditions under which 
standards-based improvement had been effective in leading to improved teaching 
and learning. They concluded that the original theory of how standards-based 
improvement works — with standards, assessments and accountability — needed 
to be expanded to capture the actual changes that distinguished successful from 
unsuccessful standards-based improvement efforts in practice (see Figure 1 below). 




(NRC, 1999a, p. 20) 



This expanded theory of action of standards-based educational improvement starts 
with alignment of standards, assessment and accountability requirements. For the 
theory to work in practice, both information and responsibility must be distributed 
throughout the system. Information about what students are expected to know and 
be able to do (standards), information about how this knowledge and ability will be 
measured (assessments), and information about how the results of such measures 
wiU be used (for accountability and to guide improvement) must be available to 
everyone — students, teachers, policymakers and the public. Likewise, responsibility 
for making use of information to improve educational quahty must be shared 
throughout the system. 

The expanded theory assumes that simply aligning standards, assessment and 
accountabihty will not necessarily lead to higher levels of learning. It is not enough 
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to clarify expectations for achievement — in the form of standards and assessments 
aligned with standards — and to motivate teachers to work harder by tying 
rewards and punishments to test results. A comprehensive educational improvement 
system must go further, providing educators (and everyone else) with both quality 
information about the kinds of educational practices that result in higher levels of 
student learning and opportunities for teachers to acquire the knowledge, skills and 
abilities they need to implement such practices (NRC, 1999a, p. 20-21). 

The NRC model offers a vision of educational improvement that fits well with the 
EFF vision of standards-based improvement in adult education. Therefore, we have 
used the NRC model as a general guide in developing assessments to be used for 
accountabihty for adult performance on the EFF Standards. For standards-based 
improvement to work for both accountability and learning results in adult basic 
education, the first step is aligning assessments and accountability with standards 
that define what is important for adults to learn. 



Aligning Standards and Assessments: 

Making Sure We Measure What We Value 

One of the most fundamental problems with current accountability policies in adult 
basic education is that there is very little connection between curriculum content or 
learning goals of adults and the content of the tests used to report learning gains for 
accountabihty. In effect, our accountabihty systems value what we are able to measure 
rather than measuring what we value. To improve this situation, we need standards 
that define learning results that matter and assessments that are 
aligned with these standards. Under these conditions, accountabihty 
assessments can provide information that is meaningfully 
connected to instructional program quahty. 

How do we define the learning results that matter most in adult 
basic education? The EFF initiative approached this task through 
a long-term, participatory and consensus-budding process that 
engaged educators, policymakers and content specialists in 
answering the complex question: What do adults need to know and 
be able to do to carry out their roles and responsibilities as workers, 
parents and family members, and citizens and community 
members? 

The NRC’s 1999 report. How People Learn: Brain, Mind, Experience and School, 
characterizes the changes in the demands of adult life in the last quarter of the 
20th century as follows: 

The skiU demands for work have increased dramatically, as has 
the need for organizations and workers to change in response to 
competitive workplace pressures. Thoughtful participation in the 
democratic process has also become increasingly complicated as the 
locus of attention has shifted from local to national and global 



What adults need to know and be 
able to do to meet the demands 
of life in the 21st century involves 
a broader range of skills, more 
complex skills and deeper content 
knowledge — as expressed in the 
EFF Content Standards — than 
current assessments are designed 
to measure. 
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concerns. Above all, information and knowledge are growing at a 
far more rapid rate than ever before in the history of humankind. 

(NRC, 1998, p. 3) 

Not only are the skill demands for adults increasing; the time available for learning 
skills is limited. Most students who come to adult education have a long-term goal of 
earning a general equivalency diploma (GED) or high school diploma. Most also come 
with very specific short-term goals: passing the driver’s license exam, finding a job, 
helping their children do better in school, learning enough English to participate fully 
in American life. Roughly half of the adults served in U.S. adult basic education 
programs are in English as a second language (ESE) classes. Once they achieve their 
short-term goals, adults leave the learning program, returning when another learning 
need emerges in their lives. This pattern, which EEE calls “just-in-time-learning,” has 
been confirmed by a study of persistence supported by the National Center for the 
Study of Adult beaming and Eiteracy (Comings, Parella & Soricone, 1999). 

We developed the EEE Standards to help adult literacy programs better address these 
very specific adult learning needs. By defining standards that focus on the integrated 
skill processes adults must use, EEE helps students develop a foundation they can use 
right away to carry out their daily responsibilities as parents, citizens and workers. 

The 16 EEE Content Standards are markedly distinct from the definitions of basic skills 
or the lists of competencies that are the foundation for standardized tests used in adult 
basic education. Therefore, existing standardized tests do not align well with the EEE 
Standards. The lack of good assessments to measure applied skills is not a problem of 
adult education alone. The development of new assessments in both K-12 and adult 
education has not kept pace with current research-based knowledge of human 
learning, development and performance. According to assessment expert Robert 
Mislevy, “(I)t is only a slight exaggeration to describe the test theory that dominates 
educational measurement today as the application of 20th-century statistics to 
19th-century psychology.” (1993, p. 19) 

What adults need to know and be able to do to meet the demands of life in the 21st 
century involves a broader range of skills, more complex skills and deeper content 
knowledge — as expressed in the EEE Content Standards — than current assessments 
are designed to measure. Therefore, ahgning assessments with the EEE Standards is not 
simply a matter of selecting among existing standardized tests. Instead, the EEE 
Assessment Eramework can guide assessment developers and adult educators in 
creating the kinds of assessments that will measure performance on the EEE Standards. 
Specifically, the EEE Assessment Eramework: 

• Identifies key dimensions of adult performance that reflect current understandings 
of adult learning and development. 

• Identifies “benchmarks,” or behavioral markers, that can be used to mark levels of 
performance that describe dimensions of proficiency on each standard. 

• Uses the key dimensions of performance and levels of performance for each standard 
to create design specifications for assessments. 
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Key Dimensions of Adult Performance: 

The EFF Continuum of Performance 

We looked to theory and research on how adults learn and develop expertise to identify 
key dimensions of adult performance. Specifically, we rehed on the key characteristics 
of expert performance synthesized by the NRC Committee on 
Developments in the Science of Learning to define a coherent set 
of theory-based dimensions of performance that teachers could 
use to pay attention to key characteristics of learning that build 
expertise and enhance transfer of knowledge. (NRC, 1996) 

To make sure that the dimensions we defined would make sense 
to teachers, we reviewed descriptions of adult performance on 
the standards prepared by teachers in our field research sites in 
relation to the key characteristics of expert performance. Through this process, 
we developed this set of four theory-based, user-friendly dimensions to define a 
developmental continuum of performance on the EFF Standards: 

■ Structure of Knowledge Base 

The hterature on expertise and transfer of knowledge asks us to think not only 
about what and how much someone knows (the number of facts, procedures, concepts 
and so on) but also how that knowledge is organized. The goal is to ensure that, as 
knowledge about a particular domain or skill grows, the structure of the knowledge 
base also develops, becoming increasingly coherent, principled, useful and goal- 
oriented. This means that what someone knows — at whatever level of knowledge — 
is organized for efficient retrieval and application in everyday life. 

This dimension of performance focuses on three aspects of learning with 
understanding: 

• A growing knowledge base of facts organized around core principles, concepts and 
procedures 

• The ability to use a growing range of cognitive strategies to organize information 
into meaningful patterns 

• The ability to use a growing range of metacognitive strategies to monitor and reflect 
on learning and performance. 

■ Fluency of Performance 

Drawing on the qualities of expert performance described above, we define fluency as 
the level of effort required for an adult to retrieve and apply relevant knowledge. 

■ Independence of Performance 

An important indicator of an adult’s increasing skill is the extent to which he or 
she needs direction or guidance in using that skill. We use DeFabio’s definition of 
independence for this dimension that emphasizes the importance of metacognition: 

“an individual’s ability to select, plan, execute and monitor his or her own performance 
without reliance on the direction of others.” (DeFabio, 1994, p. 42) Points along a skill 
development continuum for this dimension of performance would reflect a decreasing 



Taken together, these four dimensions 
of performance provide a coherent, 
research-based picture of teaming 
and developing proficiency on the 
Standards. 
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need for assistance in carrying out these metacognitive functions, whether the adult 
is acting alone or in collaboration with others. 

■ Range of Conditions for Performance 

This dimension focuses both on how well an adult can use a skill and whether he or 
she can “transfer” learning from one context to another. Our concept of range includes 
variables related to both task and context. These variables include the type as well 
as the number of tasks and contexts in which an adult uses a skill, the degree of 
familiarity or unfamiliarity of a task or context, and the complexity of the task. 

Taken together, these four dimensions of performance provide a coherent, research- 
based picture of learning and developing proficiency on the Standards. This model of 
adult performance guided our design of tasks, collection of data on adult performance 
on the EFF Standards and performance-level descriptions. (For details on field data 
collection and analyses, see Equipped for the Future/National Reporting System Data 
Collection Project, 2000-2001: An interim report on the development of the Equipped for 
the Future Assessment Framework, Equipped for the Future Assessment Consortium, 
July 2002). 

Levels of Performance: 

Defining Proficiency on the EFF Standards 

The Continuum of Performance for each EFF Standard provides a picture of what 
developing expertise looks like for adult learners using the standard. To create this 
picture, we arrayed along a continuum all the data that our teacher/ researchers pro- 
vided about learner performance. Then we reviewed this 

continuum, identifying commonalities across multiple reports of learner 
performance at a particular range on the continuum and looking for places 
in the data where multiple reports described a quahtative change in learners’ 
ability to use the standard. We marked such quahtative changes as a level. 
Thus, each EFF performance level defines a key threshold in adult learning 
and development — characterized by observable changes in proficiency. 

In some cases we noticed five qualitative changes in learner performance 
along the continuum. In these cases, we created five levels. In other cases, we 
noticed six qualitative changes. In these cases, we created six levels. 

The descriptions of performance at each EFF performance level focus on observable 
(and therefore measurable) changes in proficiency from one level to the next. The 
performance levels are like individual rungs on a ladder of increasing proficiency. 
Because we want to draw attention to the benchmarks (features of performance) that 
indicate change from one level to the next, we also do not repeat aspects of learner 
performance that remain the same from level to level. 



Each EFF performance level 
defines a key threshold in adult 
learning and development — 
characterized by observable 
changes in proficiency. 
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There are six performance levels on the Read With Understanding 
Continuum of Performance. Each level provides a detailed 
description of what proficient performance of the Standard looks 
like at that level. These six performance levels do not make up the 
full lifelong learning continuum for Read With Understanding. 

Rather, they reflect the range of adult learner performance data 
that came from the wide variety of instructional settings for both 
adult basic education and ESL that we included in our field 
research. 

Each performance-level description for each EEE Standard is 
divided into three parts: 

1. Defining the Standard 

The performance-level description starts with the components 
of performance of the Standard. These components define the 
content Standard and they remain the same at each level of 
performance. Repeating the definition of the Standard serves as 
a reminder that the integrated skill process is constant across all 
levels, from novice to expert levels of performance. What changes 
from level to level is the growth and complexity of the underlying 
knowledge base and the resulting increases in fluency and 
independence in using the Standard to accomphsh an increasing 
range and variety of tasks. 

The definition of the Standard is a useful tool for communicating to adults and 
their teachers the essential features of the construct, or set of targeted abilities, for 
each Standard. By making it clear how the skill process is defined (or “unmasking the 
construct,” as described by Gitomer & Bennett, 2002), adult learners are better able to 
articulate their own learning goals for improving proficiency and teachers are better 
able to focus learning and instructional activities that build toward the goal of 
increasing ability to use the Standard to accomplish everyday activities. Here is how 
the Standard is defined for performance levels in Read With Understanding. 



The performance levels are like 
individual rungs on a ladder of 
increasing proficiency. 




Read With Understanding Performance Level 1 



At this level, adults are able to: 

Read With Understanding 

• Determine the reading purpose. 

• Select reading strategies appropriate to the purpose. 

• Monitor comprehension and adjust reading strategies. 

• Analyze the information and reflect on its underlying meaning. 

• Integrate it with prior knowledge to address reading purposes. 
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2. Performance Indicators 

Defining Key Knowledge, Skills and Strategies 

Next, each performance-level description defines key knowledge, skills and strategies to 
be mastered for proficient performance at that level. These are the primary behavioral 
indicators (or benchmarks) of proficient performance on the standard at each level. 
This hst is thus of central importance for designing assessments to measure 
performance on the Standard. 

Because the performance levels are designed primarily as guides for assessment and 
not as a curriculum framework, the list does not specify details of knowledge, skills and 
strategies that might be studied and taught. Nonetheless, the list can serve as a way of 
identifying instructional objectives for each level and developing criteria for placement 
of learners in instructional levels. It also can be used by curriculum developers and 
teachers to set instructional objectives for each level and to develop more detailed 
curricula or learning plans that will help learners develop these abilities, meet the 
criteria and move on to the next level of expertise. 

Here is how key knowledge, skills and strategies are defined for Read With 
Understanding Performance Level 1 : 



Use Key Knowledge, Skills and Strategies 



Adults performing at this level are able to: 

• Recognize everyday words or word groups in short, simple text by decoding 
letter-sound correspondence, isolating and saying first/last sounds, naming 
pictures to isolate and say initial sounds, sounding out words by segmenting 
words into separate sounds and syllables, combining or blending sounds, 
recognizing simple rhyming word patterns, or recalling oral vocabulary and 
sight words. 

• Demonstrate familiarity with concepts of print, letter shapes, letter names 
and sounds (individual consonants and vowels, digraphs and blends), and 
common vocabulary. 

• Monitor accuracy of decoding and word recognition using various strategies 
such as rereading or making word lists. 

• Recall prior knowledge to assist in understanding information in the text. 
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Defining Fluency, Independence and Ability To Perform in a Range of Settings 

Next, each performance-level description defines the fluency, independence and abihty 
to perform in a range of settings expected for proficient performance on the Standard 
at each level. With the descriptions of key knowledge, skills and strategies, these 
descriptions serve as the primary behavioral indicators (benchmarks) of proficient 
performance at each level. These descriptions also provide a basis for designing 
learning, instruction and assessment that is appropriate to that level. Here is how 
fluency, independence and abihty to perform in a range of settings are defined for 
Read With Understanding Performance Level 1: 



Show Fluency, Independence and Ability To Perform in a 
Range of Settings 



Adults performing at Level 1 can read and comprehend words in short, 
simple texts slowly and with some effort but with few errors, to independently 
accomplish simple, well-defined and structured reading activities in a few 
comfortable and familiar settings. 



3. Examples of Proficient Performance 

Finally, each performance-level description includes a short list of examples of the 
purposeful applications (activities) that an adult who is proficient at that level can 
accomplish. This list of examples is illustrative and not exhaustive. The hst springs 
from actual reports from teachers of what adults can use the Standard to accomphsh. 
These real-world examples are useful to adults and their teachers in making concrete 
the purpose and need for attaining increasing proficiency in performance on the 
Standard. The list of real-world accomplishments also provides guidance for selecting 
and designing content for instructional materials and assessments. Here are examples 
of proficient performance for Read With Understanding Performance Level 1: 



Examples of Proficient Performance 



Adults performing at Level 1 can read with understanding to accomplish a 

variety of goals, such as: 

• Reading a grocery list and recognizing words and prices in a store ad to make 
decisions about what to buy. 

• Reading personal names and addresses to make an invitation list. 

• Reading product names and quantities to fill a purchase order. 

• Reading names and office numbers to distribute interoffice mail to the correct 
locations. 

• Reading personal information prompts to accurately fill out simple 
applications, registration forms and so on. 

• Reading product and store names or symbols on signs and storefronts to 
identify places to shop. 

• Reading months, days and dates on a personal calendar to identify and enter 
important events. 
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We need to start the assessment 



design process with a clear 
understanding of the cognitive 



Design Specifications for Assessments: 

Collecting — and Interpreting — Evidence of Performance 

Our approach to designing assessments of performance on the EFF Standards has 
been influenced by the principles of evidence-centered assessment design developed 
at Educational Testing Service (ETS) by Mislevy, Steinberg & Almond (1999) and 
by principles of model-based performance assessment developed by Baker (1998) 
and colleagues at the federally funded National Center for Research on Evaluation, 
Standards and Student Testing (CRESST), as well as by the broader trends and 
developments in cognitive science and educational measurement 
summarized in the NRC Committee on the Foundations of Testing 
pubhcation, Knowing What Students Know: The science and design 
of educational assessment (2001). 



processes underlying proficient 
performance. 



The introduction to this NRC report notes that assessments, no matter 
how technically sound, are always imprecise to some degree. In the words 
of the NRC report, “an assessment result is an estimate, based on samples 
of knowledge and performance from the much larger universe of everything that a 
person knows and can do” (NRC, 2001, p. 37, emphasis in the original). In essence, 
assessments are tools that permit us to draw reasonable inferences about what a person 
knows and can do — and thus assessment is a process of “reasoning from evidence” 
(NRC, 2001, p. 38, citing Mislevy, 1994, 1996). 



To ensure that the inferences we draw from assessment results are sound, we need to 
collect and interpret good evidence. What counts as good evidence depends upon what 
we beheve about the nature of the competence we are attempting to measure in the 
assessment. In other words, we need to start the assessment design process with a clear 
understanding of the cognitive processes underlying proficient performance. This 
understanding or model of cognition will shape the design of observations to collect 
evidence of performance. The nature of evidence collected and the model of cognition 
that informs the collection of evidence will in turn shape the methods used to interpret 
the evidence as an indication of performance on the model of cognition. 



The 2001 NRC report identifies three basic elements of good assessment: 



• Cognition: the definition of the aspects of achievement to be assessed 

• Observation: the definition of tasks that will be used to collect evidence about 
performance ability 

• Interpretation: the methods used to analyze the evidence resulting from the 
performance on the tasks 
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These three elements are portrayed as an “assessment triangle” (see Figure 2) to 
emphasize the interrelationships among all three elements. According to the 2001 
NRC report, no assessment can be designed and 
implemented without consideration of each of the 
three elements in the triangle. Making each element 
explicit and making sure that all three elements work 
well together is the key to good assessment. 



Starting with Cognition: 

Know What You’re Measuring 

In applying the principles of good assessment to the 
design of assessments of performance on the EFF 
Standards, we start with a model of cognition, which 
represents learning and development in a particular 
domain of performance, such as reading. Each EEE 
Content Standard defines an adult learning domain, which includes: 



Figure 2. The Assessment Triangle 



Observation 



Interpretation 




Cognition 



(NRC, 2001, p. 39) 



• The EFF Standard and its components of performance, which name and define the 
domain as a particular skill process 



• The four dimensions of the EFF Continuum of Performance, which describe the 
general model of performance in each domain 

• The performance-level descriptors, which identify the specific behavioral indicators 
on each of the four dimensions of performance (increasing depth and structure of 
the knowledge base and increasing fluency, independence and ability to perform in a 
range of conditions) 

Taken together, the EFF Standard, the dimensions of performance and the 
performance-level descriptors for the standard represent a theory-based, empirically 
derived and field-tested model of how adults develop expertise in the domain defined 
by the EFF Standard. 

From Cognition to Observation: 

Choose Appropriate Ways To Measure Performance 

The next step in designing assessments of performance on the EFF 
Standards involves the observation corner of the triangle, which represents 
a set of beliefs about the kinds of tasks or situations that will best afford 
opportunities to observe and collect evidence on the important knowledge, 
skills and strategies of performance identified under cognition. 

Because the EFF Standards are complex skill processes, we believe performance 
assessments are the most suitable method for observing and collecting evidence of 
performance on the standards. Performance assessments simulate the conditions in 
which key skills are used in real-world activities — optimizing the ability of the 
assessments to measure learning that will transfer to daily life. 



Performance assessments 
simulate the conditions in 
which key skills are used in 
real-world activities — 
optimizing the ability of the 
assessments to measure 
learning that will transfer 
to daily life. 
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Other forms of assessment may provide a partial measure of an EFF Standard. 
Assessments with a selected-response format, for example, have the advantage of ease 
of administration and easily achieved reliability in scoring. But they fall short in 
capturing comprehensive evidence of performance on tasks involving the purposeful 
and integrated apphcation of a skills process as defined in the EFF Content Standards. 

Over the years, there has been skepticism about using performance assessments for 
high-stakes testing. Indeed, performance assessments can be difficult and expensive 
to develop. Scorers require expertise and training. And educational attitudes and 
practices may stand in the way of the positive changes in teaching and learning that 
performance assessments can spur. Yet interest in performance assessments remains 
strong, largely due to the opportunities they provide to measure complex thinking 
and applied skills — as well as the way they can direct the attention of teachers and 
learners to important learning goals (such as integrating learning, solving problems 
and communicating effectively). 

Taking a model-based approach makes it possible to overcome many of the drawbacks 
of performance assessment. A strong model of cognition — coupled with tight 
specifications for assessment design linked directly to that model — enables assessment 
developers to use the specifications continually to produce new performance tasks and 
scoring rubrics. This approach also enables teachers and others to use the model of 
cognition to incorporate important content into instructional activities and to monitor 
student learning (Baker, 1998). 

ETS’s evidence-centered design and CRESST’s model-based performance assessment 
focus on laying out a detailed template or paradigm that serves as the basis for 
structuring tasks and collecting and evaluating complex performance evidence in 
ways that support inferences or conclusions about what a person is able to do. 

In our adaptation of these two approaches, we created a template of generic design 
specifications for accountability assessments. An example of the generic design 
specifications for Performance Fevel 3 of the EFF Standard Read With Understanding 
is on the following page. 
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Performance Level 3. Design Specifications for Accountability 
Assessment: Read With Understanding 



Characteristics of Assessment Tasks 

Actions (Performance Goals): Reading and understanding printed text to make a 
decision or accomplish an action 

Type of Texts: Multiple pages of simply written and simply structured continuous 
text on high-interest subjects (excerpts from informational texts geared at lower- 
level readers, such as a simple handbook about job benehts) 

Content: Common, high-interest vocabulary 
Enviromnent: A range of comfortable and familiar settings 
Duration: No more than two hours 

Materials: Selected or prepared text(s), simple question/answer sheets, 
documentation forms 

Knowledge and Strategies Assessed at This Levei 

• Decoding and recognizing mostly everyday words, but also some new and 
polysyllabic words, by drawing on prior familiarity with content and oral 
vocabulary, breaking words into parts, applying pronunciation rules, and 
adjusting reading pace 

• Familiarity with common, high-interest content (such as job benehts); related 
vocabulary; and pronunciation rules 

• Monitoring and enhancing comprehension by using a range of simple strategies 
such as recalhng, restating, rephrasing, explaining the content of the text or 
using simple examples 

• Activating prior knowledge and combining it with new information 

Evidence 

Observations: Oral reading; oral answers to questions about reading 
Work Products: Short written answers to questions about reading 
Task Structure: Tasks are simple, well dehned and highly structured 



From Observation to Interpretation: 

Making Valid, Reliable Inferences About Performance 

Performance assessments produce evidence — observations and work 
products — that enable us to make inferences about an adult’s ability to 
perform on the skills that constitute the standards. 

To interpret the evidence and make a judgment about cognition, the EFF 
specifications for assessment design include a template for the scoring 
rubrics that should be used to evaluate the performance evidence 
produced by tasks. Here is an example of a task-specific scoring rubric 
for Read With Understanding, Performance Level 3: 



The EFF Content Standards, 
performance descriptions, 
specifications for assessment 
design and scoring rubrics are 
tightly aligned, which will enable 
assessment developers to 
create reliable performance 
assessments. 
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Scoring Rubric Template for Accountability Assessment: 
Read With Understanding Performance Level 3 


Unable 
to Score 


Beginning 


Proficient 


Advanced 




• Decodes and recognizes 
some but not all new words 


• Decodes and recognizes mostly 
everyday words, but also some 
new and polysyllabic words, by 
drawing on prior familiarity with 
content and oral vocabulary, 
breaking words into parts, 
applying pronunciation rules, 
and adjusting reading pace 


• Easily decodes and 
recognizes new words 




• May not always 
successfully use 
monitoring strategies 


• Monitors and enhances 
comprehension by use of a range 
of simple strategies such as 
recalling, restating, rephrasing, 
explaining the content of the text 
or using simple examples 


• Easily uses monitoring 
strategies and adjusts 
strategies as needed 




• May have difficulty 
combining prior knowledge 
with new information 


• Activates prior knowledge and 
combines it with new information 


• Combines prior knowledge 
with new information and 
begins to draw inferences 
based on combined 
information 




• Decodes and recognizes 
words slowly and hesitantly 
at first; may need more 
than one attempt, but 
completes the task with 
some direction and support 
and with some errors 


• Decodes and recognizes words 
accurately and easily; completes 
task with little direction or support 
and with few errors 


• Decodes and recognizes 
words without hesitation 
comfortably; completes 
task without direction or 
support and without major 
errors 



Together, the generic design specifications and the task-specific scoring rubrics provide 
the methods and tools for designing assessments, collecting evidence of performance 
and drawing inferences about an adult’s level of proficiency on an EFF Standard. The 
generic design specifications are based on the knowledge and strategies to be assessed 
at each performance level — and the specifications are linked directly to evidence of 
performance from the assessment task. 

The EFF Content Standards, performance descriptions, specifications for assessment 
design and scoring rubrics are tightly ahgned, which will enable assessment developers 
to create reliable performance assessments. Still, because of individual differences 
among assessors, all observations and interpretations are fallible to some degree. 
Therefore, we recommend building in cross checks and redundancy in our methods of 
scoring assessments and using assessment results to make decisions about proficiency 
levels. Specifically, we recommend: 
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• At least three assessments on every EFF Standard. Multiple measures are always 
preferable to determine an adult’s performance level in accountability assessments 

• Two independent ratings of performance evidence. Reliability of scoring is critical 
for high-stakes uses of EFF accountability assessments 

Tests and assessments used in adult education need to provide results that are accurate 
for all types of learners in the system. Ensuring fairness of tests and assessments 
used for any purpose requires careful attention to the consequences of test use and 
interpretation. Fairness means looking for evidence that patterns of assessment results 
reveal bias in favor of one group of learners over another. If such evidence is found, 
the assessment should be examined closely to identify (and eliminate) any features 
unrelated to the key knowledge and skills that the assessment is meant to measure 
that can be shown to bias results in favor of one group of test takers over another. 

These and other aspects of administering, scoring and reporting EFF assessments for 
accountability are described in more detail in Section 3. 



How Can Assessments of the EFF Standards Support 
Both Accountability and Instructional Improvement? 

Almost everyone would agree in theory that the ultimate goal of holding adult 
education programs accountable for achieving learning results is to make high-quality 
learning opportunities more broadly available. But accountability mechanisms that 
simply identify programs that succeed in order to sustain them and identify those 
that fail in order to eliminate them do not necessarily provide information that can 
be used for program improvement. Achieving continuing improvement in adult basic 
education and reaching higher levels of adult learning is possible when the content of 
accountabihty assessments is aligned with standards for curriculum and instruction, 
when the meaning of assessment results is clearly and easily understood by all, and 
when results are fair to all. 

This is what we have set out to accomplish in designing assessments for the EFF 
Standards and in developing the components of the Read With Understanding 
Assessment Prototype. 

As the NRC (1999a) makes clear in its expanded model of standards-based educational 
improvement (see Figure 1 on page 8), however, professional development and 
improved teaching are critical to supporting higher levels of learning as well. Here 
too, ahgnment is paramount to success. 

• First, aligning standards that reflect important knowledge and skills with assessments 
that fuUy capture this content and provide a complete picture of the learning that 
occurs in adult basic education. 

• Second, aligning professional development with research-based knowledge about 
effective instructional practices. This ahgnment fosters the environment teachers 
need to change and improve instructional practices. 
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Transparenq^ and information sharing are additional requirements for the effective 
functioning of standards-based improvement. AU stakeholders — adult learners, 
teachers, program managers, policymakers and the pubhc — need to understand what 
the results of accountabihty tests mean and how these results can be used to indicate 
program quahty and to guide instructional improvement. This means that the results 
of assessments used for accountability need to be transparent — easily and clearly 
understood by all. A strong and well-defined cognitive process model communicates 
clearly what adults should know and be able to do in key content areas. 

Transparency makes it possible to use the assessment triangle — cognition, observation 
and interpretation — to build assessment tasks and scoring rubrics that reflect clearly 
defined models of learning and development. Curriculum and instruction also can be 
built around these same models of learning and development, while professional 
development for teachers can focus on effective strategies for teaching to clearly 
defined models of learning and development. This is why “unmasking the construct” 
is so important in high-stakes assessment. 

The EFF levels of performance illustrate how transparency works to improve teaching 
and learning as well as accountability: 

• Learners (alone or with the teacher) can use the performance levels to identify where 
they are and where they need to go. 

• Teachers can use the performance levels to guide instruction by focusing on key 
knowledge, skills and strategies and to guide observations and informal evaluations 
of student progress in performance on the standard. 

• Program directors can work with teachers to use the performance levels as a guide 
to identify areas for staff development for effective instruction. 

Finally, states and programs can use the information provided by accountability 
assessments as a reality check on their opportunities for adult learners. While fairness 
of test results depends on the features of assessments and the interpretation of results, 
fairness cannot be achieved without a fair distribution of teaching and learning 
opportunities — a process that transparency and information sharing facilitate. 

In Sections 2 and 3, we will show how we have applied cognitive theory and developed 
assessments to meet accountability demands in adult basic education and ESF 
programs. Section 2 provides details on the EFF Assessment Prototype for the EFF 
Read With Understanding Standard. Section 3 provides practical guidance for using the 
EFF Assessment Prototype for assessment and accountability reporting of learning on 
the EFF Standards. 
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SECTION 2 

The Accountability Assessment Prototype 
for Read With Understanding 



T he EFF Assessment Prototype is a standardized, alternative assessment 
system designed for use in adult education. The prototype includes a 
collection of tools and guidance to help adult education programs and 
states assess educational gains in reading to meet the requirements of 
the U.S. Department of Education’s National Reporting System. These tools and 
guidelines also can be adapted and used to develop assessments on other EFF 
standards. 

The tools in the EFF Assessment Prototype include performance-level descriptions 
that describe key features of proficiency, assessment task specifications to guide 
the development of performance-based assessments for each performance level, 
a scoring rubric and model assessment task for each level, and an initial set of 
30 performance-based assessments that are designed to meet National Reporting 
System requirements for standardized alternative assessment and reporting of 
educational gains in reading. The assessment prototype is a demonstration of the 
EFF approach to assessment for standards-based accountabihty and educational 
improvement. As such, the prototype can serve as a model that test developers 
can use to expand the range of assessment tools available for accountability and 
standards-based improvement. 

A Secure, Password -Protected Web Site 

The initial set of 30 secure assessment items (five at each of the six performance 
levels) for Read With Understanding is available on our secure, password-protected 
Web site. Access to these assessment items will be provided to state adult education 
agency staff, adult education program staff, and others who request and are approved 
for access by the EFF Assessment Consortium. 



In this section you will 

learn about: 

■ The relationship 
between the EFF 
performance levels and 
the U.S. Department of 
Education’s National 
Reporting System 

■ The six performance 
levels for the EFF Read 
With Understanding 
Standard 

■ Assessment design 
specifications, model 
assessment tasks and 
scoring rubrics for 
each of the six 
performance levels 



EFF Performance Levels Align With the National Reporting System 

In developing EFF performance levels, our goal was to make it possible for programs to 
report learner progress in using EFF Standards on the Educational Functioning Fevels 
in the National Reporting System developed by the Division of Adult Education and 
Fiteracy, Office of Adult and Vocational Education, U.S. Department of Education. 

In our earlier work, we engaged 100 practitioners in five states in collecting learner 
performance data so we could develop EFF performance levels that truly reflected adult 
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development. In 2000, the Division of Adult Education and Literacy at the U.S. 
Department of Education began to support our research for a separate but related 
purpose of its own. Many states had been using these Educational Eunctioning Levels 
for a purpose for which they were not intended — as benchmarks for knowledge and 
skills that adult learners should be expected to master to move from one level to the 
next. The U.S. Department of Education’s Division of Adult Education and Literacy 
asked us to use our research data to enrich and enlarge the existing Educational 
Eunctioning Level descriptions, with the goal of making them robust enough to serve 
the purposes states wanted them for — as benchmarks of standardized assessment 
of adult learning and reporting of adult learning. 

To map the EEE Read With Understanding Standard to the National Reporting 
System, we first validated our descriptions of knowledge, skills and abilities at each 
performance level of the EEE Continuum of Performance. Then, we compared the 
content of each EEE performance level to the related knowledge, skills and abilities 
described in the National Reporting System Educational Eunctioning Levels. Table 1 
shows the results of this effort. Each EEE performance levels marks an exit point for a 
National Reporting System Educational Eunctioning Level. This means that when an 
adult has demonstrated mastery of the proficiency in reading defined in an EEE 
performance level, the adult can be considered to have moved up one Educational 
Eunctioning Level. 



Table 1: Correspondences Between EFF and the National Reporting System 


EFF Continuum of 
Performance... 


...defines an exit point for 
the National Reporting 
System Adult Basic 
Education (ABE) 
Educational Functioning 
Level 


...defines an exit point for 
the National Reporting 
System English as a 
Second Language (ESL) 
Educational Functioning 
Level 


Read With Understanding 
Performance Level 1 


Beginning ABE Literacy 


Beginning ESL 


Read With Understanding 
Performance Level 2 


Beginning Basic Education 


Low Intermediate ESL 


Read With Understanding 
Performance Level 3 


Low Intermediate Basic 
Education 


High Intermediate ESL 


Read With Understanding 
Performance Level 4 


High Intermediate Basic 
Education 


Low Advanced ESL 


Read With Understanding 
Performance Level 5 


Low Adult Secondary 
Education 


High Advanced ESL 


Read With Understanding 
Performance Level 6 


High Adult Secondary 
Education 
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The EFF performance levels are relevant to the assessment and reporting 
requirements of the National Reporting System for both adult basic education and 
ESF. Adults who meet the EFF Standards at Performance Fevel 1, for example, are 
ready to exit Beginning ABE Fiteracy or Beginning ESF, the corresponding National 
Reporting System Educational 
Functioning Fevels. For reporting 
purposes, they can be classified at 
the Beginning Basic Education or 
Fower Intermediate ESF levels. 

Next, we’ll turn our attention to 
the specific details of the Read 
With Understanding performance 
levels and our assessment 
prototype for this EFF standard. 

The EFF Standard Read 
With Understanding 

Our assessment prototype 
builds on the EFF Standard 
Read With Understanding and 
on six performance levels we’ve 
developed for the Read With 
Understanding Continuum of 
Performance. We’ll describe the 
six performance levels beginning 
on page 27. First, though, recall 
the definition of the EFF 
Standard Read With 
Understanding (at right). 

This definition — developed 
with input from expert 
instructors, reading specialists 
and researchers in adult basic literacy and ESF — characterizes reading proficiency for 
adults, who read to accomphsh tasks in the real world. 

The focus of the Read With Understanding Standard, in fact, is the ability to perform 
tasks that require effective interaction with print. What happens between determining 
the reading purpose and achieving that purpose is a complex cognitive and behavioral 
process that involves a reader interacting with a text and a reading activity. The 
proficient adult reader must have and be able to apply knowledge, skills, abilities 
(especially strategic abilities) and other characteristics (including motivation) to 
interact with printed materials (in various formats and at various levels of complexity) 
to accomplish a range of reading activities, each of which is shaped by its social and 
cultural context. 



To fulfill responsibilities as parents/family members, 
citizens/community members, and workers, 
adults need to be able to: 



STANDARD 



Read With 
Understanding 




EFF ASSESSMENT CONSORTIUM 



25 




Improving Performance, Reporting Results 



Section 2: The Accountability Assessment Prototype for Read With Understanding 



Guide to Reading Performance-Level Descriptions 

Performance-Level Descriptions on the Read With 
Understanding Continuum of Performance 

As we discussed in Section 1 on page 14, performance-level descriptions for EFF 
Standards include: 



LEVEL INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

A list of knowledge, skills and strategies to be mastered characterizes performance 
on the standard at each level of proficiency. This list of learning and instructional 
objectives provides the foundation for developing: 

• Criteria for placing adults in reading instructional levels 

• Specifications for assessments to monitor their progress in acquiring the reading 
knowledge, skills and strategies associated with each level 

• Specifications for assessments to measure attainment of proficiency in reading 
with understanding at each level 

■ Fluency, Independence and Ability To Perform in a Range of Settings 

A description of aspects of fluency, independence and range of performance 
characterizes skills to be mastered at each level of proficiency. Like the list of key 
knowledge, skills and strategies, this description helps teachers and assessors 
design appropriate learning, instruction and assessment for each level. 



The EFF Standard 



The components of 
performance define the 
Standard. The definition of 
the Standard remains the 
same at each levei of the 
continuum, serving as a 
constant reminder to 
teachers and adult learners 
of the integrated skills the 
Standard represents. 



EXAMPLES OF PROFICIENT PERFORMANCE 



This list of activities that adults can accomplish at each level of proficiency is 
illustrative, not exhaustive. The examples make clear and concrete to adult 
learners and their teachers the increasing sophistication of purposes and tasks 
that can be achieved as knowledge and skills increase — and they motivate adults 
to reach for higher levels of proficiency. The hst of real-world accomplishments 
also provides guidance for selecting and designing instructional materials and 
assessments. 
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EFF Standard 



How adults at Level 1 

Read With Understanding 

• Determine the reading 
purpose. 

• Select reading strategies 
appropriate to the purpose. 

• Monitor comprehension 
and adjust reading 
strategies. 

• Analyze the information 
and reflect on its 
underlying meaning. 

• Integrate it with prior 
knowledge to address 
reading purpose. 



PERFORMANCE LEVEL 1 

Read With Understanding 

Exit point for National Reporting System Beginning ABE Literacy and Beginning ESL 



LEVEL 1 INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

Adults performing at Level 1 can: 

• Recognize everyday words or word groups in short, simple text by decoding 
letter-sound correspondence, isolating and saying first and last sounds, 
naming pictures to isolate and say initial sounds, sounding out words by 
segmenting words into separate sounds and syllables, combining or 
blending sounds, recognizing simple rhyming word patterns, or recalling 
oral vocabulary and sight words. 

• Demonstrate familiarity with concepts of print, letter shapes, letter names 
and sounds (individual consonants and vowels, digraphs and blends), and 
common vocabulary. 

• Monitor accuracy of decoding and word recognition using various strategies, 
such as rereading or making word lists. 

• Recall prior knowledge to assist in understanding information in the text. 



■ Show Fluency, Independence and Ability To Perform 
in a Range of Settings 

Adults performing at Level 1 can read and comprehend words in short, 
simple texts slowly and with some effort but with few errors, to independently 
accomplish simple, well-defined and structured reading activities in a few 
comfortable and familiar settings. 



LEVEL 1 EXAMPLES OF PROFICIENT PERFORMANCE 



Adults performing at Level 1 can read with understanding to accomplish a variety 

of goals, such as: 

• Reading a grocery list and recognizing words and prices in a store ad to make 
decisions about what to buy. 

• Reading personal names and addresses to make an invitation list. 

• Reading product names and quantities to fill a purchase order. 

• Reading names and office numbers to distribute interoffice mail to the correct 
locations. 

• Reading personal information prompts to accurately fill out simple 
applications, registration forms and so on. 

• Reading product and store names or symbols on signs and storefronts to 
identify places to shop. 

• Reading months, days and dates on a personal calendar to identify and 
enter important events. 
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EFF Standard 



How adults at Level 2 

Read With Understanding 

• Determine the reading 
purpose. 

• Select reading strategies 
appropriate to the purpose. 

• Monitor comprehension 
and adjust reading 
strategies. 

• Analyze the information 
and reflect on its 
underlying meaning. 

• Integrate it with prior 
knowledge to address 
reading purpose. 



PERFORMANCE LEVEL 2 

Read With Understanding 

Exit point for National Reporting System Beginning Basic Education 
and Low Intermediate ESL 



LEVEL 2 INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

Adults performing Level 2 can: 

• Decode and recognize everyday, simple words in short, simple text by breaking 
words into parts, tapping out/sounding out syllables, applying pronunciation 
rules, using picture aids, and recalling oral vocabulary and sight words. 

• Demonstrate famibarity with simple, everyday content knowledge and 
vocabulary. 

• Locate discrete items of information in simplified text. 

• Monitor and enhance comprehension using various strategies, such as 
rereading, restating, copying and rephrasing text; making a list of new words; 
or using a simplified dictionary. 

• Recall prior knowledge to assist in selecting texts and in understanding the 
information they contain. 



■ Show Fluency, Independence and Ability To Perform 
in a Range of Settings 

Adults performing at Level 2 can read and comprehend words in small blocks of 
simple text, slowly but easily and with few errors, to independently accomplish 
simple, well-defined and structured reading activities in a range of comfortable 
and familiar settings. 



LEVEL 2 EXAMPLES OF PROFICIENT PERFORMANCE 



Adults performing at Level 2 can read with understanding to accomplish a variety of 

goals, such as: 

• Reading aloud a picture book with very simple text to a young child. 

• Reading a short narrative about a community concern to identify and think 
about personal community issues. 

• Reading about entry-level job duties to decide whether to apply. 

• Reading simple greeting cards to choose an appropriate card for a friend. 

• Reading a simple chart about job benefits to figure out if hospitalization is 
covered. 

• Reading utility bills to understand how and when to pay them. 

• Reading short narratives about immigrant experiences to reflect on and learn 
about personal heritage. 

• Reading the weather forecast in the newspaper to decide on appropriate clothes 
for a weekend trip. 
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EFF Standard 



How adults at Level 3 

Read With Understanding 

• Determine the reading 
purpose. 

• Select reading strategies 
appropriate to the purpose. 

• Monitor comprehension 
and adjust reading 
strategies. 

• Analyze the information 
and reflect on its 
underlying meaning. 

• Integrate it with prior 
knowledge to address 
reading purpose. 



PERFORMANCE LEVEL 3 

Read With Understanding 

Exit point for National Reporting System Low Intermediate Basic Education 
and High Intermediate ESL 



LEVEL 3 INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

Adults performing at Level 3 can: 

• Decode and recognize most everyday and some unfamiliar words in short to 
medium-length text by drawing on content knowledge and oral vocabulary, 
breaking words into parts, applying pronunciation rules, and adjusting reading 
pace. 

• Demonstrate famibarity with common, high-interest content knowledge and 
related vocabulary. 

• Locate important information in simple text using some simple strategies. 

• Monitor and enhance comprehension by using a range of simple strategies, such 
as recalling, restating, rephrasing, explaining the content of the text or using 
simple examples. 

• Actively apply prior knowledge to assist in understanding information in texts. 



■ Show Fluency, Independence and Ability To Perform 
in a Range of Settings 

Adults performing at Level 3 can quickly and accurately read and comprehend 
words and word groups in multiple pages of simple text to independently 
accomplish simple, well-defined and structured reading activities in a range of 
comfortable and familiar settings. 



LEVEL 3 EXAMPLES OF PROFICIENT PERFORMANCE 



Adults performing at Level 3 can read with understanding to accomplish a variety of 

goals, such as: 

• Reading about a company’s job benefits to make decisions about personal 
choice of benefits. 

• Reading a short story about how cultural differences can lead to conflict to 
reflect on and make decisions about personal issues. 

• Reading a minimum-wage poster to determine if a job wage is legal. 

• Reading a short story about losing a job to reflect on the ways job loss can affect 
family relationships. 

• Reading citizenship application procedures to help someone decide whether to 
pursue citizenship. 

• Reading housing rental ads to compare housing options and make a decision 
about which house is better for a family. 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



EFF Standard 



How adults at Level 4 

Read With Understanding 

• Determine the reading 
purpose. 

• Select reading strategies 
appropriate to the purpose. 

• Monitor comprehension 
and adjust reading 
strategies. 

• Analyze the information 
and reflect on its 
underlying meaning. 

• Integrate it with prior 
knowledge to address 
reading purpose. 



PERFORMANCE LEVEL 4 

Read With Understanding 

Exit point for National Reporting System High Intermediate Education 
and Low Advanced ESL 



LEVEL 4 INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

Adults performing at Level 4 can: 

• Recognize unfamiliar and some specialized words and abbreviations using word 
analysis or inference. 

• Demonstrate familiarity with everyday and some specialized content knowledge 
and vocabulary. 

• Locate important information in text using a wide range of strategies. 

• Monitor and enhance comprehension using a range of strategies, such as posing 
and answering questions, trial and error, and adjusting reading pace. 

• Organize information using some strategies, such as recall, restatement, simple 
sequencing and simple categorization. 

• Actively apply prior knowledge to assist in understanding information in texts. 



■ Show Fluency, Independence and Ability To Perform 
in a Range of Settings 

Adults performing at Level 4 can read and comprehend a variety of texts at an 
appropriate pace and with good comprehension to independently accomplish 
structured reading activities in a variety of familiar settings. 



LEVEL 4 EXAMPLES OF PROFICIENT PERFORMANCE 



Adults performing at Level 4 can read with understanding to accomplish a variety of 

goals, such as: 

• Reading fast food nutrition charts to choose a meal that is low in fat. 

• Reading brief newspaper editorials on opposing sides of a subject of interest to 
clarify a personal opinion on the subject. 

• Reading newspaper advice columns to stimulate thinking about personal issues. 

• Reading TV Guide to determine if specific movies are appropriate for 
children. 

• Reading information about labor unions to make a decision about joining a 
union. 

• Reading a magazine about typical behavior for toddlers to figure out how to 
deal with a two-year-old’s tantrums. 

• Reading a brochure from a health chnic to learn about signs of depression and 
helpful tips for deahng with it. 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



EFF Standard 



How adults at Level 5 

Read With Understanding 

• Determine the reading 
purpose. 

• Select reading strategies 
appropriate to the purpose. 

• Monitor comprehension 
and adjust reading 
strategies. 

• Analyze the information 
and reflect on its 
underlying meaning. 

• Integrate it with prior 
knowledge to address 
reading purpose. 



PERFORMANCE LEVEL 5 

Read With Understanding 

Exit point for National Reporting System Low Adult Secondary Education 
and High Advanced ESL 



LEVEL 5 INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

Adults performing at Level 5 can: 

• Recognize and interpret abbreviations and specialized vocabulary. 

• Demonstrate familiarity with everyday and some specialized content knowledge 
and vocabulary and with paragraph structure and document organization. 

• Locate important information, read identified sections for detail and determine 
missing information using a wide range of strategies. 

• Monitor and enhance comprehension using a wide range of strategies. 

• Organize and analyze information and refiect upon its meaning using a range 
of strategies such as classification, categorization and comparison/contrast. 

• Evaluate prior knowledge against new information in texts to enhance 
understanding of the information. 



■ Show Fluency, Independence and Ability To Perform 
in a Range of Settings 

Adults performing at Level 5 can read and comprehend dense or multipart texts at 
an appropriate pace and with good comprehension to independently accomplish 
structured, complex reading activities in a variety of familiar and some novel 
settings. 



LEVEL 5 EXAMPLES OF PROFICIENT PERFORMANCE 



Adults performing at Level 5 can read with understanding to accomplish a variety of 

goals, such as: 

• Reading over-the-counter medicine labels to choose the right product 
for a sick child. 

• Reading a magazine article about home Internet connection providers to 
analyze the options described and select a provider. 

• Reading information about advertising techniques and analyzing the ways 
advertisers persuade consumers to buy their products to become better 
informed about a purchase. 

• Reading a campus safety brochure to be aware of a school’s rules, regulations 
and resources available for students. 

• Reading information from the Occupational Safety and Health Administration 
about noise exposure to solve a problem at work. 

• Reading a self-help book about family finances to find ways to save money. 

• Reading information about voter eligibility to decide if one is eligible to vote in 
an upcoming election. 

• Reading instructions from a Web site for job seekers to find information and 
advice on effective resume writing. 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



EFF Standard 



How adults at Level 6 

Read With Understanding 

• Determine the reading 
purpose. 

• Select reading strategies 
appropriate to the purpose. 

• Monitor comprehension 
and adjust reading 
strategies. 

• Analyze the information 
and reflect on its 
underlying meaning. 

• Integrate it with prior 
knowledge to address 
reading purpose. 



PERFORMANCE LEVEL 6 

Read With Understanding 

Exit point for National Reporting System High Adult Secondary Education 



LEVEL 6 INDICATORS 



■ Use Key Knowledge, Skills and Strategies 

Adults performing at Level 6 are able to: 

• Recognize and interpret terms, signs, symbols, acronyms and abbreviations. 

• Demonstrate familiarity with extensive specialized content knowledge and 
vocabulary and with the organization of long, complex prose and complex 
documents. 

• Use a wide range of strategies to guide reading of long texts. 

• Locate both directly stated and implied important information. 

• Monitor and enhance comprehension using a wide range of strategies, such as 
brainstorming and question formulation techniques. 

• Organize and analyze information and reflect upon its meaning using a wide 
range of strategies, such as applying relevant information to multiple scenarios, 
summarizing, and drawing “big picture” conclusions and generalizations from 
detailed reading. 

• Integrate prior knowledge with new information in texts to develop deep 
understanding of the information. 



■ Show Fluency, Independence and Ability To Perform 
in a Range of Settings 

Adults performing at Level 6 can read and comprehend long, complex texts at an 
appropriate pace and with good comprehension to independently accomplish 
structured or unstructured complex reading activities in a variety of familiar and 
novel settings. 



LEVEL 6 EXAMPLES OF PROFICIENT PERFORMANCE 



Adults performing at Level 6 can read with understanding to accomplish a variety of 

goals, such as: 

• Reading information about financial aid for higher education to decide whether 
to apply for loans and to understand options available if applying for aid. 

• Reading a brochure on workplace medical benefits to distinguish differences 
between types of plans available and choose the best personal family plan. 

• Reading newspaper editorials that take opposite stands on the same issue 
and decide which argument is more persuasive to develop a personal position 
on the issue. 

• Reading a consumer guide about long-distance telephone services to choose a 
personal home service. 

• Reading a journal article on childhood bullying to get ideas about how to cope 
with a child’s bullying behavior. 

• Reading data sheets on material safety to get guidance about safely handhng 
toxic materials in the workplace. 

• Reading a company’s employee handbook to get up-to-date information about 
the company’s employment leave policies. 
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Design Specifications for Accountability Assessments 

Our assessment design specifications for the EFF Standard Read With Understanding 
provide detailed guidelines for developing tasks and scoring rubrics that can be used 
as accountability measures for each level of performance. These guidehnes are useful 
to assessment developers who want to design formal assessments for accountability 
purposes and can also be used by teachers who want to create informal assessments 
to use for instructional purposes. 

Specifically, we build on the descriptions of fluent and independent use of knowledge 
and strategies at each level of the EFF Continuum of Performance to describe the 

characteristics of high-quality assessment tasks in terms of: 

Actions or Performance Goals 
Content 
Environment 
Duration 
Materials 

Structure and Process 



For each of the six performance levels, our design specifications also describe: 

• Knowledge and strategies that should be assessed 

• Types of evidence — work products and observations — that can be used to 
evaluate performance 

• Guidelines for scoring performance in the form of a scoring rubric template 
that describes the quahties of a proficient performance as well as the qualities of 
performances that fall just below (beginning) and just above (advanced) proficiency 

A proficient performance on an assessment task provides evidence that an adult 
learner is performing at the level that the task was designed to measure. A beginning 
(below proficient) level of performance may indicate that the adult learner should be 
tested on a task designed to measure performance at the next lower level. An advanced 
(above proficient) level of performance may indicate that the adult learner should be 
tested on a task at the next higher level. 
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Design Specifications and Scoring Rubric Templates 
for Accountability Assessments 

PERFORMANCE LEVEL 1: Read With Understanding 



Characteristics of Assessment Tasks 

Action — Performance Goal. Gathering information 
from print to get ready to make a decision or 
accomplish an action 

Type of Text. Short, simple, high-interest texts 
(simple personal lists, personal names and addresses, 
grocery ads, signs, packages) 

Content. Simple, famihar vocabulary 

Environment. One or more comfortable, familiar 
settings 

Estimated Time To Complete Task. Up to one hour 
Materials. Specific text(s), documentation forms 

Knowledge and Strategies Assessed at 
Performance Level 1 

• Recognizing everyday words or word groups 
in simple text by decoding letter-sound 
correspondence, isolating and saying first and last 
sounds, naming pictures to isolate and say initial 
sounds, sounding out words by segmenting words 
into separate sounds and syllables, combining or 
blending sounds, recognizing simple rhyming word 
patterns, or recalling oral vocabulary and sight 
words 

• Demonstrating familiarity with concepts of print, 
letter shapes, letter names and sounds (individual 
consonants and vowels, digraphs and blends), and 
common vocabulary 

• Monitoring accuracy of decoding and word 
recognition using various strategies, such as 
rereading or making word lists 

• Recalling prior knowledge to assist in understanding 
information in the text 



Evidence 

Observations. Oral reading; oral answers to 
questions about reading 

Work Products. Word lists, oral or graphic word 
matches on simple lists, forms, advertisements, and 
other simple print contexts 

Task Structure 

Tasks are simple, well defined and highly structured 
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Scoring Rubric Template — EFF Performance Level 1 

Unable 

to Score Beginning Proficient Advanced 





• Recognizes some 
but not all letter/sound 
relationships, some 
syllables, and words or 
word groups 


• Recognizes letter-sound 
relationships, some syllables, and 
words or word groups (two-to- 
three-word items), using such 
strategies as isolating and saying 
first and last sounds, naming 
pictures to isolate and say initial 
consonants, sounding out words 
by segmenting words into 
separate sounds or syllables, 
combining or blending sounds, 
recognizing simple rhyming word 
patterns, using oral vocabulary, 
and memorizing sight words 


• Easily recognizes letter- 
sound relationships, 
syllables, and some words 
or word groups 




• May not always accurately 
list new words or reread 
words in a new print 
context 


• Effectively monitors retention of 
letter-sound and word recognition 
by accurately listing new words or 
rereading words in more than one 
print context 


• Easily and accurately lists 
new words or rereads 
words in and out of order 
and in multiple print 
contexts 




• May have difficulty 
linking prior knowledge 
to demands of task 


• Easily recalls prior knowledge of 
topic to assist in understanding 


• Uses prior knowledge of 
topic to make Inferences 
that support recognition of 
a wide range of new words 




• Decodes and recognizes 
words slowly and hesitantly 
at first; may need more 
than one attempt, but 
completes the task with 
some direction and 
support, and with some 
errors 


• Decodes and recognizes simple 
words slowly with some effort, but 
completes task with little direction 
or support and with few errors 


• Decodes and recognizes 
words without hesitation 
comfortably, completes 
task without direction or 
support and without 
major errors 
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Model Assessment Task 

PERFORMANCE LEVEL 1: Read With Understanding 

Delivering Interoffice Mail 



Characteristics of Assessment Tasks 

Action — Performance Goal. Read names and office 
numbers to simulate delivery of five interoffice mail 
envelopes to the employees’ office locations 

Type of Text. 

• Office location labels (room numbers) 

• List of employee names (one to two syllables) and 
their office locations, each designated by a letter 
followed by three numbers 

• Sample pages, each depicting an interoffice mail 
delivery envelope with an employee’s first and last 
name (one to two syllables) 

Content. Mostly common one- to two-syUable 
decodable employee names (Pam Ford, Josh Rust, 
etc.); letter/number combinations to designate office 
locations (AlOl, Cl 15, D102, etc.); terms on task 
materials (interoffice mail, name, office, building) 

Environment. Familiar, comfortable environment 
such as the classroom 

Estimated Time To Complete Task. 15 minutes 
Materials. 

• Assessor Observation Form 

• List of 12 employees’ names and their office 
locations (Figure 1) 

• A sample interoffice mail envelope with employee/ 
recipient names written on it for the assessor to use 
for demonstration purposes (Figure 2 DEMO) 

• Five additional pictures showing interoffice mail 
envelopes (Figures 3-7) 

• 12 office location labels (Label 1 - 12) 

• Tape recorder 



Knowledge and Strategies Assessed at 
Performance Level 1 

Proficient performance requires: 

• Recognizing words or word groups (two to three 
words) in simple, noncontinuous text by decoding 
letter-sound correspondence, isolating and saying 
first/last sounds, naming pictures to isolate and say 
initial consonants, sounding out words by 
segmenting words into separate sounds/syllables, 
combining or blending sounds, recognizing simple 
rhyming word patterns, and recalling oral 
vocabulary and sight words 

• Familiarity with concepts of print, letter shapes 
(uppercase and lowercase letters, consonants and 
vowels), letter names and sounds (individual 
consonants and vowels, digraphs and blends), and 
common vocabulary 

• Monitoring accuracy of decoding and word 
recognition using strategies such as rereading or 
making word lists 

• Recalling prior knowledge of the topic to assist in 
understanding 

Additional Knowledge and Strategies 
That May Affect Performance 

• Some familiarity with concept and function of 
work-related interoffice mail 

• Some familiarity with the use of letter/number 
combinations on room numbers 

• Some familiarity with conventions of hsting names 
(in this case, last names are listed first) 

Evidence 

Observations. Assessor Observation Form 

Work Products. 

• Completed distribution of mail to office locations 

• Tape recording of student reading list of names 
aloud 
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Model Assessment Task: Delivering Interoffice Mail 
Step-by-Step Procedures for Administration 

This task is administered to individual learners. While there is no time limit to complete the task, it should 
require no more than 15 minutes to complete. You may repeat any of the steps as needed, but you must speak 
in English and read the steps exactly as they are written in the scripts each time. You also may model any 
requirement of the task to help clarify, as needed. 



Step 1 

Explain the task requirements and expectations: 

“This task will show how well you can read first and last 
names and match them to office numbers.” 

Step 2 

Give the learner the list of names and office addresses 
and ask the learner to read it silently first and then out 
loud. 

“Here is the list of employees’ names and their office 
numbers (Figure 1). The last names are listed before the 
first names. Read the list of names silently to yourself. Tell 
me when you have finished.” 

Turn on the tape recorder. 

“I have turned on the tape recorder. Read the list of names 
out loud.” (Record any mistakes in decoding and reading 
aloud names on Assessor Observation Form.) 

Turn off tape recorder. 

Step 3 

Explain the next steps in the task: 

“Today, you will practice delivering the mail. Here is an 
example of what you will do. Here is a picture of an 
envelope addressed to Jim Allen.” 

(Point to the name on the envelope of Figure 2.) 

“Here is a list of the employees and their office addresses.” 
(Point to Figure 1.) 

“I will use the list to find Jim Allen’s office address.” 

(Use your finger to scan the list and point to Jim Allens office 
address on the list.) 

“Jim Allen’s office is AlOl.” 



Place the 12 office labels (Labels 1 through 12) in front of 
the learner. 

“These 12 office labels are the office locations where you 
wiU be delivering the mail. I have to deliver mail to Jim 
Allen’s office, AlOl. Once I find Jim Allen’s office from 
these labels, I can deliver his mail.” 

Eind the label that reads “AlOl” and place Figure 2 on top 
of that label. 

“You will deliver the remaining envelopes.” 

(Hold up Figures 3 through 7.) 

Ask if there are any questions. Wait briefly to see if the 
learner understands the task; if not, answer questions. 
You may repeat the demonstration if requested. 

Step 4 

Give the learner the illustrations of envelopes (Figures 3 
through 7). Note the time the learner begins the task on 
the Assessor Observation Form. 

“Now you can start delivering the mail. You can take as 
much time as you need.” 

Step 5 

Observe as the learner begins to work on the task. 

You may repeat the instructions if the learner did not 
understand using the example in Step 3. Use the Assessor 
Observation Form to record comments about learner 
performance. 

Step 6 

When the learner finishes, record the number of 
correctly delivered envelopes on the Assessor 
Observation Form, collect all materials and note the 
amount of time the learner has taken to complete 
the task. 
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Scoring Rubric: Delivering Interoffice Mail — EFF Performance Level 1 

Unable 

to Score Beginning Proficient Advanced 





• Decodes and accurately 
reads aloud 1 7 or fewer 
name words 


• Decodes and accurately reads 
aloud at least 1 8 of the 24 name 
words 


• Decodes and accurately 
reads aloud 1 9 or more 
name words 




• Is able to read and match 
up fewer than three of 
the full names on the 
envelopes with the names 
on the Staff List 


• Locates at least three of the full 
names on the envelopes and 
match those with names on the 
Staff List 


• Is easily able to locate all 
five of the full names on the 
envelopes and match those 
with names on the Staff 
List 




• Is able to identify the office 
numbers of fewer than 
three staff members to 
correctly deliver the mail 
to those offices 


• Is able to identify the office 
numbers of at least three staff 
members to correctly deliver the 
mail to those offices 


• Quickly identifies the office 
numbers of all the staff in 
order to correctly deliver 
mail to all offices 




• Is able to complete the task 
of delivering the mall for 
fewer than three full names 


• Is able to complete the task of 
delivering the mail for at least 
three full names 


• Is readily able to complete 
the task of delivering the 
mail for all full names 




• Needs continual direction 
and support. Needs more 
than one attempt to 
understand the task or is 
not able to complete It 


• May recognize and match names 
with office numbers slowly, with 
some effort and repetition (going 
between envelope and list) but 
completes the task with little 
direction or support and with 
few errors 


• Completes the task without 
direction and support 
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Assessor Observation Form (Group): Delivering Interoffice Mail 
Step-by-Step Procedures for Administration 

ASSESSOR NAME DATE 

Part 1 . 

Circle any name words that the learner does not read aloud accurately. 



1. Allen 


7. Ford 


13. Kolt 


19. Rust 


2. Jim 


8. Pam 


14. Ned 


20. Josh 


3. Camp 


9. Fox 


15. Metis 


21. Selk 


4. Vern 


10. Todd 


16. Bob 


22. Meg 


5. Chand 


11. Holm 


17. Pratt 


23. Wells 


6. Frank 


12. Liz 


18. Rick 


24. Kong 



Total number of name words accurately read aloud 

Part 2. 

Notes on matching of full names to office addresses. 

Number of correctly delivered envelopes 

Use this notes section to record any observations regarding the learner’s performance, such as: 

• How much direction, repetition or support the learner needs 

• Speed, ease of and strategies used for word recognition 

• Any obvious signs of comfort or frustration with the task 

• Ability to complete the task within the time allowed 
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Delivering Interoffice Mail 



Figure 1. Staff List 



Staff List 


EMPLOYEE 


OFFICE 


Allen, Jim 


A101 


Camp, Vern 


A110 


Chand, Frank 


A111 


Ford, Pam 


A102 


Fox, Todd 


D105 


Holm, Liz 


C101 


Kolt, Ned 


B109 


Metts, Bob 


B104 


Pratt, Rick 


C106 


Rust, Josh 


D103 


Selk, Meg 


B108 


Wells, Kong 


D112 
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Delivering Interoffice Mail 



Figure 2 Demo. 
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Delivering Interoffice Mail 



Figure 4 




Figure 5 
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Delivering Interoffice Mail 



Figure 6 
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Delivering Interoffice Mail 



Labels 



Label 1 

A101 


Label 7 

B109 


Label 2 

A110 

j 


Label 8 

B104 

L 


Label 3 

A111 


Label 9 

Cl 06 


Label 4 

A102 


Label 10 

D103 


Label 5 

D105 


Label 11 

BIOS 


Label 6 

Cl 01 




Label 12 

D112 
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Delivering Interoffice Mail 



Answer Key (Samples) 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Design Specifications and Scoring Rubric Templates 
for Accountability Assessments 

PERFORMANCE LEVEL 2: Read With Understanding 



Characteristics of Assessment Tasks 

Action — Performance Goal. Gathering information 
from print to get ready to make a decision or 
accomplish an action 

Type of Text. Short, small blocks of continuous 
text with simple sentence structure, predictable or 
repetitive narrative elements (such as a children’s 
book), and/or phonetically regular words 

Content. Simple, familiar vocabulary 

Environment. A range of comfortable, familiar 
settings 

Estimated Time To Complete Task. Up to one hour 

Materials. Specific text(s) as well as other necessary 
tools, documentation forms or question/answer 
sheets 

Knowledge and Strategies Assessed at 
Performance Level 2 

• Decoding and recognizing everyday, simple words 
in short, simple text by breaking words into parts, 
tapping out or sounding out syllables, applying 
pronunciation rules, using picture aids, and 
recalling oral vocabulary and sight words 

• Demonstrating familiarity with simple, everyday 
content knowledge and vocabulary 

• Locating discrete items of information in simphfied 
text 

• Monitoring and enhancing comprehension using 
various strategies such as rereading, restating, 
copying and rephrasing text; making a list of new 
words; or using a simplified dictionary 

• Recalling prior knowledge to assist in selecting texts 
and in understanding the information they contain 



Evidence 

Observations. Oral reading; oral answers to 
questions about reading 

Work Products. Short written or oral statement of 
meaning of each page or section of text 

Task Structure 

Tasks are simple, well defined and highly structured 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Scoring Rubric Template — EFF Performance Level 2 

Unable 

to Score Beginning Proficient Advanced 





• Decodes and recognizes 
some but not all words 


• Decodes and recognizes everyday, 
simple words, using such strategies 
as stressing vowel and consonant 
sounds, tapping out or sounding 
out syllables, using picture prompts, 
using a picture or other simplified 
dictionary 


• Easiiy decodes and 
recognizes words; reads 
aioud with expression 
appropriate to the meaning 




• May have difficulty 
accurately locating 
needed information 


• Accurately locates needed 
information 


• Easiiy iocates ail needed 
information 




• Sometimes repeats 
rather than rephrases; 
explanations may be 
incomplete 


• Effectively monitors and 
enhances comprehension by 
restating and rephrasing small 
blocks of continuous text to 
explain its meaning 


• Uses a variety of strategies 
to monitor and enhance 
comprehension and can 
independentiy adjust 
strategies as needed 




• May have difficulty linking 
prior knowledge to 
demands of task 


• Easily recalls prior knowledge 
to assist in choosing appropriate 
reading materials and in 
comprehending small blocks 
of continuous text 


• Uses prior knowiedge to 
make inferences across 
and comprehend multipie 
biocks of text 




• Reads slowly and hesitantly 
at first; may need more 
than one attempt, but 
completes the task with 
some direction and 
support and with some 
errors 


• Reads slowly and with effort, but 
completes task with little direction 
or support and with few errors 


• Reads without hesitation 
comfortably, completes 
task without direction or 
support and without major 
errors 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Model Assessment Task 

PERFORMANCE LEVEL 2: Read With Understanding 

Reading Utility BiUs 



Characteristics of Assessment Tasks 

Action — Performance Goal. Read two utility bills 
(a telephone bill and an electric biU) to gather 
information about how and when to pay them 

Type of Text. Simplified telephone biU and simplified 
electric bill, each containing smaU blocks of simple 
text (company name and address, customer name 
and address, amount, and date due) arrayed in several 
locations on a page; two sets of simply structured 
questions 

Content. Common, high-interest content and mostly 
simple, everyday words including “question words” — 
who, what, how much — and doUar amounts, but 
also a few terms less common in everyday speech 
( amount, due, company ) 

Environment. Famihar, comfortable environment 
such as the classroom 

Estimated Time To Complete Task. 20 minutes 
Materials. 

• Simplified version of a telephone bill (Figure 1) 

• Simplified version of an electric biU (Figure 2) 

• Short-answer written response form (Answer 
Sheet 1) 

• Oral response form (for assessor to ask questions 
and record responses) (Answer Sheet 2) 

• Assessor Observation Form 

• Tape recorder 

• Pencils, pens, highlighters 



Knowledge and Strategies Assessed at 
Performance Level 2 

Proficient performance requires: 

• Decoding and recognizing everyday, simple 
words by breaking words into parts, tapping out or 
sounding out syUables, applying pronunciation 
rules, using picture aids, and asking for “hints” 

• FamUiarity with common knowledge and simple, 
everyday vocabulary 

• Monitoring and enhancing comprehension by using 
some simple strategies such as rereading, restating, 
recopying and rephrasing text; making a fist of new 
words; using a simplified dictionary; or asking for 
help 

• Recalling prior knowledge to assist in choosing 
appropriate texts and comprehending their meaning 

Additional Knowledge and Strategies That 
May Affect Performance 

• Some familiarity with concepts of utilities and 
paying for services provided 

• Some familiarity with format of bills 

Evidence 

Observations. Documentation of reading aloud on 

Assessor Observation Form 

Work Products. 

• Written responses on Answer Sheet 1 

• Learner responses (written by teacher) to questions 
on Answer Sheet 2 

• Tape recording of oral reading and performance 
answering questions 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Model Assessment Task: Reading Utility Bills 
Step-by-Step Procedures for Administration 

The oral response section of this task will be administered to individual learners; the written response section may 
be administered to individuals or a group of learners. There is no time hmit on this task but it should require no 
more than 20 minutes to complete. You may repeat the reading of any of the steps as needed, but you must speak 
in English and read the steps exactly as they are written in the script. You also may model any requirement of the 
task to help clarify, as needed. 



Step 1 

Explain the task requirements and expectations: 

“For this task, you wiU be asked to read some information 
on two bills and answer questions about how much is 
owed for each one and when they need to be paid.” 

Step 2 

Give out copies of the telephone hill — Figure 1 — along 
with Answer Sheet 1. Hold up the telephone hill and 
Answer Sheet 1 and point to the appropriate places on 
the hill and Answer Sheet while saying: 

“Please look first at the telephone bill. Read this biU silently 
to yourself. When you have finished reading, please write 
answers to the questions about this bUl on Answer Sheet 1. 
You may refer to the biU as you write your answers and you 
can make notes or highlight on the bUl or on the answer 
sheet. 

“Are there any questions before we begin?” 

Step 3 

Instruct the learners to begin the task: 

“Please start reading the telephone bill and then complete 
Answer Sheet 1. When you have finished, please bring your 
Answer Sheet and the copy of the phone bUl to me.” 

Step 4 

When the learner completes written responses, pass out 
copies of the electric bill — Figure 2. Hold up the electric 
bill and say: 

“In a moment I am going to ask you to read this electric 
bill out loud to me. I will record you as you read. Then I 
wUl ask you three questions about that biU. I wUl write 
down what you say. Every time you answer a question I 
wUl ask you to point to the part of the bUl where you 
found your answer. 

“Are there any questions before we begin?” 

Step 5 

Turn on tape recorder, pick up Assessor Observation 



Form and ask the learner to begin, pointing to the 
appropriate corners of the bill and motioning down the 
page while giving directions: 

“I am turning on the tape recorder now. Please read the 
whole electric bill out loud to me, starting at the top with 
all the words in one corner, then all the words in the other 
corner, then moving down the page. You may begin when 
you are ready.” 

While learner reads, document the reading aloud on the 
Assessor Observation Form. 

Step 6 

When learner has finished reading aloud, pick up Answer 
Sheet 2 and ask learner to respond to questions orally: 

“Please answer the following questions. After you answer, 
please point to the place on your biU where you found the 
answer. 

“How much is the electric bill?” 

When learner answers and points, write down the answer on 
Answer Sheet 2. If learner does not point, you may prompt 
him or her, saying, “Point to the place on the bill where you 
found your answer. 

“What is the due date?” 

When learner answers and points, write down the answer on 
Answer Sheet 2. If learner does not point, you may prompt 
him or her, saying “Point to the place on the biU where you 
found your answer. 

“What is the name of the company that sent the biU?” 

When learner answers and points, write down the answer on 
Answer Sheet 2. If learner does not point, you may prompt 
him or her, saying “Point to the place on the biU where you 
found your answer.” 

Step 7 

Turn off the tape recorder. Record your observations on 
the Assessor Observation Form. Collect all task materials 
and record the approximate time the learner has taken to 
complete the task. 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Scoring Rubric: Reading Utility Bills — EFF Performance Level 2 

Unable 

to Score Beginning Proficient Advanced 





• Decodes and recognizes 
some words but has 
trouble with others 

• Sometimes may use 
strategies like recognizing 
sight words, such as 
“name,” “date,” “number” 


• Decodes and recognizes most 
commonly used words (“name,” 
“date,” “telephone number,” “total”) 
as well as some less commonly 
used words (“amount,” “due,” 
“company”) 


• Decodes and recognizes all 
words easily 




• Strategies for 
monitoring and enhancing 
comprehension need 
improvement; may restate 
or rephrase information 
inaccurately; can correctly 
answer fewer than five 
questions 


• Effectively monitors comprehension 
by restating or rephrasing 
information about due dates, total 
amounts and names of companies 
well enough to correctly answer five 
out of seven questions 


• Shows superior monitoring 
and comprehension skills 
by locating information 
quickly and restating or 
rephrasing it well enough 
to correctly answer all 
questions 




• Reads slowly and 
hesitantly; may need more 
than one attempt but 
completes the task with 
some direction and support 


• Reads slowly and with effort but 
completes task with little direction or 
support 


• Reads comfortably without 
hesitation; completes task 
without direction or support 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading Utility Bills 



Figure 1: Telephone Bill 



QWEST Customer Eva Merino 

346 Montgomery Avenue Account Number 509-234-5678 

Spokane, WA 99204 



SUMMARY OF ACCOUNT 

Balance $ 0.00 

Total Amount Due $ 70.00 

Due Date December 14, 2002 



For Questions Call 1 - 800 - 321-3612 



Figure 2: Electric Bill 



AVISTA ELECTRIC 

567 Boone Avenue 
Spokane, WA 99205 



Customer Eva Merino 

Account Number 234567 



Service from October 15 to November 15, 2002 



Amount Due $ 35.00 



Due Date November 30, 2002 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading Utility Bills 



Answer Sheet 1: Telephone Bill 



NAME 



DATE 



Write answers to the questions about the telephone bill. You may refer to the bill as you 
write your answers and you can make notes or highlight on the bill or on this answer 
sheet. 



What is the total amount due? 



When is the bill due? 



What is the name of the telephone company? 



What do you do if you have questions about this bill? 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading Utility Bills 



Answer Sheet 2: Electric Bill 



(For Assessor Only) 

NAME DATE 

Write down learner’s oral responses to these questions about the electric bill. 

How much is the electric bill? 



What is the due date? 



What is the name of the company that sent the bill? 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading Utility Bills 



Answer Key for Answer Sheet 1: Telephone Bill 



NAME 

DATE 

Write answers to the questions about the telephone 
bill. You may refer to the bill as you write your 
answers and you can make notes or highlight on 
the bill or on this answer sheet. 

What is the total amount due? $70.00 



When is the bill due? 

Possible answers: December 14, 2002 

12/14/02 



What is the name of the telephone company? QWEST 



What do you do if you have questions about this bill? 

Call 1-800-321-3612 



Answer Key for Answer Sheet 2: Electric Bill 



(For Assessor Only) 

NAME 

DATE 

Write down learner’s oral responses to these 
questions about the electric bill. 

How much is the electric bill? $35.00 



What is the due date? 

Possible answers: November 30, 2002 

11/30/02 



What is the name of the company that sent the bill? 

Avista Electric 



54 



EFF ASSESSMENT CONSORTIUM 







Improving Performance, Reporting Results 



Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading Utility Bills 



Assessor Observation Form 



NAME 



DATE 



Performance 1: Learner reads electric bill aloud. 

Decodes and recognizes words: 

• Using what strategies? 

• How fluently/easily? 

• How independently? 

• Other comments? 

Notes: 



Performance 2: Learner answers questions aloud and points to information on bill to support answers. 

• Answers correctly? 

• Points to correct information on bill? 

• How fluently/easlly? 

• How Independently? 

• Other comments? 

Notes: 



EFF ASSESSMENT CONSORTIUM 



55 





Improving Performance, Reporting Results 



Section 2: The Accountability Assessment Prototype for Read With Understanding 



Design Specifications and Scoring Rubric Templates 
for Accountability Assessments 

PERFORMANCE LEVEL 3: Read With Understanding 



Characteristics of Assessment Tasks 

Action — Performance Goal. Reading and 
understanding printed text to make a decision or 
accomplish an action 

Type of Text. Multiple pages of simply written and 
simply structured continuous text on high-interest 
subjects (excerpts from informational texts geared at 
lower-level readers, such as a simple handbook about 
on-the-job benehts) 

Content. Common, high-interest vocabulary 

Environment. A range of comfortable, familiar 
settings 

Estimated Time To Complete Task. One to two 
hours 

Materials. Selected or prepared text(s), simple 
question/answer sheets, documentation forms 

Knowledge and Strategies Assessed at 
Performance Level 3 

• Decoding and recognizing most everyday and 
some unfamihar words in short to medium-length 
continuous text by drawing on content knowledge 
and oral vocabulary, breaking words into parts, 
applying pronunciation rules, and adjusting reading 
pace 

• Demonstrating familiarity with common high- 
interest content knowledge and related vocabulary 

• Locating important information in simple text 
using some simple strategies 

• Monitoring and enhancing comprehension by 
using a range of simple strategies such as recalling, 
restating, rephrasing, explaining the content of the 
text or using simple examples 

• Actively applying prior knowledge to assist in 
understanding information in texts 



Evidence 

Observations. Oral reading; oral answers to 
questions about reading 

Work Products. Short written answers to questions 
about reading 

Task Structure 

Tasks are simple, well dehned and highly structured 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Scoring Rubric Template — EFF Performance Level 3 

Unable 

to Score Beginning Proficient Advanced 





• Decodes and recognizes 
some but not all new words 


• Decodes and recognizes mostly 
everyday words, but also some new 
and polysyllabic words, by drawing 
on prior familiarity with content and 
oral vocabulary, breaking words into 
parts, applying pronunciation rules, 
and adjusting reading pace 


• Easily decodes and 
recognizes new words 




• May have difficulty locating 
important information 


• Accurately locates important 
information 


• Easily locates important 
information 




• May not always 
successfully use 
monitoring strategies 


• Monitors and enhances 
comprehension by using a range 
of simple strategies, such as 
recalling, restating, rephrasing, 
explaining the content of the text 
or using simple examples 


• Easily uses monitoring 
strategies and adjusts 
strategies as needed 




• May have difficulty 
combining prior knowledge 
with new information 


• Activates prior knowledge and 
combines it with new information 


• Combines prior knowledge 
with new information and 
begins to draw inferences 
based on combined 
information 




• Decodes and recognizes 
words slowly and hesitantly 
at first; may need more 
than one attempt, but 
completes the task with 
some direction and 
support and with some 
errors 


• Decodes and recognizes words 
accurately and easily; completes 
task with little direction or support 
and with few errors 


• Decodes and recognizes 
words without hesitation 
comfortably, completes 
task without direction or 
support and without major 
errors 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Model Assessment Task 

PERFORMANCE LEVEL 3: Read With Understanding 

Reading a Letter From School About a Parent-Teacher 
Conference 



Characteristics of Assessment Tasks 

Action — Performance Goal. Read and understand a 
letter from a child’s school inviting the reader to a 
parent-teacher conference, answer questions about 
the information in the letter and respond to the letter 
by filling out a response card 

Type of Text. Simplified business letter with date; 
salutation; short message containing information and 
instructions concerning the why, when and where of 
the conference written in mostly simple sentences; 
basic instructions about how to respond; closing 
signature; Short Response Card with simple 
sentences, yes/no check-offs and request for signature; 
mostly simple-sentence questions (two complex 
sentences) on Answer Sheet 1 

Content. Mostly everyday, high-interest content and 
vocabulary, with some speciahzed vocabulary related 
to types of communication (“translator,” “signature”) 

Environment. Familiar, comfortable environment 
such as the classroom 

Estimated Time To Complete Task. 15 to 30 minutes 
Materials. 

• Letter from school (Figure 1) 

• Seven questions on form (Answer Sheet 1 ) 

• Response Card (Answer Sheet 2) 

• Reading Strategies Observation Form 

• Pens, pencils, highlighters, dictionaries (bilingual 
and English dictionaries) 



Knowledge and Strategies Assessed at 
Performance Level 3 

Proficient performance requires: 

• Decoding and recognizing mostly everyday words, 
but also some new and polysyllabic words, by 
drawing on prior familiarity with content and oral 
vocabulary, breaking words into parts, applying 
pronunciation rules, and adjusting reading pace 

• Familiarity with common, high-interest content 

• Monitoring and enhancing comprehension by 
using a range of simple strategies such as recalling, 
restating, rephrasing, explaining the content of the 
text or using simple examples 

• Activating prior knowledge and combining it with 
new information 

Additional Knowledge and Strategies That 
May Affect Performance 

• Some familiarity with concept of and protocols 
associated with parent-teacher conferences in 
K-12 education in the United States 

• Some understanding of cultural issues surrounding 
parental involvement with children’s education and 
relationships with teachers, as appropriate 

• Some (very basic) famiharity with business letter 
format/structure 

• Some familiarity with protocols associated with 
responding to invitations 

Evidence 

Observations. Reading Strategies Observation Form 

Work Products. 

• Written answers to seven questions on Answer 
Sheet 1 

• Completed Response Card 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Model Assessment Task: Reading a Letter From School About a Parent-Teacher 
Conference 

Step-by-Step Procedures for Administration 

This task may be administered to an individual learner or to a small group of learners. While there is no time 
limit to complete the task, it should require no more than 30 minutes to complete. You may repeat the reading of 
any of the steps as needed, but you must speak in English and read the steps exactly as they are written in the 
scripts each time. You may also model any requirement of the task to help clarify, as needed. 



Step 1 

Explain the task requirements and expectations: 

“For this task, you will show how well you can read and 
understand a letter from a school about a parent-teacher 
meeting. You will answer some questions about this letter 
and fill out a card in response to this letter.” 

Step 2 

Pass out the letter from school (Figure 1). Hold up 
Figure 1 and ask learners to read the letter silently: 

“This is a letter from a school. It invites a parent to attend 
a meeting with a teacher at the child’s school. Please read 
this letter silently to yourself. You may use a dictionary to 
look up any words you are not sure of.” 

Step 3 

Pass out Answer Sheet 1. Hold up Answer Sheet 1 and 
ask learners to write answers to the questions: 

“Now pretend that you are a parent who got the letter 
you read in the mail. Using the information you just read, 
write answers to the questions on Answer Sheet 1. You 
may reread the letter and you can make notes or highlight 
on the letter or the answer sheet. In some cases, there is 
more than one right answer to a question. For some 
answers you will want to use what you already know 
about parent-teacher meetings as well as what you just 
read in the letter.” 



Step 4 

Pass out the Response Card on Answer Sheet 2. Hold up 
Answer Sheet 2. Ask learners to fill out the Response 
Card: 

“After you read this letter, you need to teU the school if 
you will be attending the meeting. To do that, please fill 
out this Response Card on Answer Sheet 2.” 

Step 5 

While the learner is working on the tasks in steps 2, 3 
and 4, observe the learner, looking for evidence of the 
use of reading strategies. Write down any observations 
on the Reading Strategies Observation Form. 

Step 6 

Collect all task materials and record the approximate 
time the learner has taken to complete the task. 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Scoring Rubric for Reading a Letter From School About a Parent-Teacher Conference — 
EFF Performance Level 3 

Unable 

to Score Beginning Proficient Advanced 





• Decodes and recognizes 
some everyday words and 
some specialized words; 
has some difficulty recalling 
and restating information 
from the letter; correctly 
answers fewer than four of 
the first five questions on 
Answer Sheet 1 


• Decodes and recognizes 
most everyday words and some 
specialized words, and accurately 
recalls and restates most information 
by correctly answering four of the first 
five questions on Answer Sheet 1 


• Easily decodes and 
recognizes everyday and 
specialized words, and 
accurateiy recalis and 
restates information by 
correctiy answering ail of 
the first five questions on 
Answer Sheet 1 




• Strategies for 
monitoring and enhancing 
comprehension need 
improvement; student is 
not observed using, or 
effectively using, strategies 
listed on Assessor 
Observation Form (for 
example, may use a 
dictionary but looks up 
every word) 


• Shows use of strategies for 
monitoring and enhancing 
comprehension; student is 
observed using at least one 
strategy listed on the Assessor 
Observation Form effectively 


• Shows superior monitoring 
and comprehension skiils 
by using at ieast three 
strategies listed on the 
Assessor Observation Form 
effectiveiy 




• Responses to questions 
6 and 7 on Answer Sheet 1 
may be incomplete or 
inappropriate; shows 
difficulty using prior 
knowledge by being unable 
to introduce information 
in answers that is not from 
the reading 


• Correctly and completely answers 
either question 6 or 7 on Answer 
Sheet 1 with little difficulty; 
shows use of prior knowledge by 
introducing appropriate information in 
the answer that is not from 
the reading 


• Easiiy and correctiy 
answers questions 6 and 7 
on Answer Sheet 1 , and 
may provide supporting 
detaii, showing use of prior 
knowledge by introducing 
appropriate information that 
is not from the reading 




• Unable to fill out Response 
Card completely or 
appropriately 


• Fills out Response Card completely 
and appropriately, with little difficulty 
and few minor errors 


• Easiiy and appropriately 
fills out Response Card 
completely 




• May need more than one 
attempt, but completes the 
task with some direction 
and support 


• Completes task with little direction or 
support 


• Completes task without 
hesitation, direction or 
support 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading a Letter From School About a Parent-Teacher Conference 



Figure 1. Letter From School 



Instructions: This is a letter from a school that invites 
a parent to attend a meeting at the child’s school with 
a teacher. Please read this letter silently to yourself. 
You may use a dictionary to look up any words you 
are not sure of. 



Oak Hill Middle School 

150 Vista Street 

November 1, 2002 
Dear Parent or Guardian: 

You are invited to come to our school for a meeting with your 
child’s teacher to discuss your child’s progress. 

The meeting will be on Thursday, November 19th from 
3 to 3:30 p.m. at our school. 

Please complete the attached card and ask your child to return 
it to his or her teacher by November 10th. Be sure to sign the 
card and tell us if you need a translator. If you cannot come at 
this time, please call the school at (509) 766-2134 to schedule 
another meeting time. 

Thank you, 

Mary Smith 
Principal 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading a Letter From School About a Parent-Teacher Conference 



Reading Strategies Observation Form 



(For Assessor Only) 

LEARNER NAME DATE 

Assessor Note: Gather as much information as you can through direct observation while the student is complet- 
ing the task. After the task is complete, you may also ask questions to heip you ascertain the reading strategies 
used and ievei of understanding of reading strategies (e.g., iimited, strong). 



Did Learner... 


Evidence 


Use a dictionary to look up words? 

EH YES EH NO 




Underline or highlight key words? 

EH YES EH NO 




Take notes on paper? 

EH YES EH NO 




Reread to improve understanding? 

EH YES EH NO 




Go back and forth between the 
ietter and the answer sheet to answer 
questions correctly? 

EH YES EH NO 




Relate what was read to things aiready 
known? 

EH YES EH NO 




other? 





Total Number of Strategies Observed: 

Levei of understanding of reading strategies: 
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Reading a Letter From School About a Parent-Teacher Conference 



Answer Sheet 1 



LEARNER NAME DATE 

Instructions: Using the information in the letter, 
write answers to the questions on this answer 
sheet. You may reread the letter and you can 
make notes or highlight on the letter or this 
answer sheet. 

1. What time will the Parent-Teacher Meeting begin? 



2. Who wrote this letter? 



3. Why are you invited to the meeting? 



4. What is the address of the school? 



5. If you cannot come to the meeting, what will you do? 



6. Write one question that you will ask your child’s teacher during the meeting. 



7. If you take your child with you to the meeting, what will he or she do while you are 
meeting with the teacher? 
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Section 2: The Accountability Assessment Prototype for Read With Understanding 



Reading a Letter From School About a Parent-Teacher Conference 



Answer Sheet 2: Response Card 



LEARNER NAME DATE 

Instructions: Pretend that you are a parent who 
got the letter you read in the mail. You need to tell 
the school if you will be attending the meeting. 

To do that, please fill out this Response Card. 



Response Card 

I will attend the Parent-Teacher Meeting at the time and day 
in the letter. 

n YES 

□ NO (I will call the school for a different day or time.) 

I will need a translator. 

n YES (What language? ) 

□ NO 

Signature of Parent or Guardian 
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Reading a Letter From School About a Parent-Teacher Conference 



Answer Key for Answer Sheet 1; Letter From School 



LEARNER NAME DATE 

Instructions: Using the information in the letter, write answers to the questions on this answer sheet. 

You may reread the ietter and you can make notes or highiight on the letter or this answer sheet 

1. What time wiil the Parent-Teacher Meeting begin? 3:00 PM 

2. Who wrote this letter? Acceptable answers: Mary Smith 

The Principal 

3. Why are you invited to the meeting? To discuss my child’s progress 

4. What is the address of the school? 150 Vista Street 

5. If you cannot come to the meeting, what will you do? Call the school to schedule another meeting time. 

6. Write one question that you wiil ask your child’s teacher during the meeting. Answers will vary; a proficient 
answer will be appropriate to the context of a parent-teacher meeting. 

7. If you take your child with you to the meeting, what will he or she do while you are meeting with the teacher? 
Answers will vary; a proficient answer will be appropriate to the context of bringing a child to a meeting. 



Answer Key for Answer Sheet 2: Response Card 



LEARNER NAME 



DATE 



Instructions: Pretend that you are a 
parent who got the letter you read in the 
mail. You need to tell the school if you 
will be attending the meeting. To do 
that, please fill out this Response Card. 
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Design Specifications and Scoring Rubric Templates 
for Accountability Assessments 

PERFORMANCE LEVEL 4: Read With Understanding 



Characteristics of Assessment Tasks 

Action — Performance Goal. Reading to locate and 
use information to make a decision or accomplish an 
action 

Type of Text. Brief but specialized authentic texts 
(want ads, simple instructions, medicine labels, 
informational tables) 

Content. May include some abbreviations and 
specialized vocabulary 

Environment. A variety of familiar or comfortable 
settings 

Estimated Time To Complete Task. One to three 
hours 

Materials. Specific text(s) as well as written 
suggestions, guides (common abbreviations list) 
or sequential instructions, visual aids such as 
informational posters or simple charts or tables for 
organizing information, question/answer sheets, 
other documentation forms 

Knowledge and Strategies Assessed at 
Performance Level 4 

• Recognizing unfamiliar (some specialized) words 
and abbreviations using word analysis or inference 

• Demonstrating familiarity with everyday and some 
specialized content knowledge and vocabulary 

• Locating important information in text using a wide 
range of strategies 

• Monitoring and enhancing comprehension using 
a wide range of strategies (such as posing and 
answering questions, trial and error, adjusting 
reading pace) 



• Organizing information using some strategies 
such as recall, restatement, simple sequencing, or 
simple categorization 

• Actively applying prior knowledge to assist in 
understanding information in texts 

Evidence 

Observations. Oral or written answers to provided 

questions about work process and strategies 

Work Products. 

• Teacher’s written notes to document completion 
of task 

• Rewritten text (for accurate decoding of 
abbreviations) 

• Short oral or written responses or answers to 
questions about task 

• Oral or written demonstrations of ability to read 
and follow sequential instructions, documented by 
assessor 

Task Structure 

Tasks may be structured and include guides and 

worksheets 
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Scoring Rubric Template — EFF Performance Level 4 

Unable 

to Score Beginning Proficient Advanced 





• Identifies some 
abbreviations and 
specialized vocabulary, 
but may confuse meanings 
and functions 


• Correctly identifies unfamiliar 
(some specialized) abbreviations 
and vocabulary by effective use of 
word analysis or inference 


• Recognizes and uses 
abbreviations and specialized 
vocabulary effectively and 
appropriately 




• Accurately locates some 
key information 


• Accurately locates important 
information using a range of 
strategies 


• Easily and accurately locates 
and begins to summarize key 
information 






• Effectively monitors and enhances 
comprehension using a range of 
strategies 






• May provide responses 
to the task that are 
incomplete or contain 
major factual or 
organizational errors 


• Effectively organizes information 
using some strategies such as 
recall, restatement, simple 
sequencing, simple categorization 






• Recalls prior knowledge 
but has difficulty combining 
it with new information 


• Accurately recalls prior knowledge 
and combines it with new 
information 


• Begins to use new information 
to reevaluate prior knowledge 




• Has some difficulty 
completing reading task 
without direction or support 


• Completes reading task with little 
direction or support and without 
major errors 


• Completes reading task with 
little or no support and with 
only a few minor errors 
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Model Assessment Task 

PERFORMANCE LEVEL 4: Read With Understanding 

George’s Annual Physical 



Characteristics of Assessment Tasks 

Action — Performance Goal. Read to locate 
important information in a dialogue to make 
decisions about healthy and unhealthy lifestyles 

Type of Text. Brief, specialized text (written dialogue) 
on the topic of healthy habits 

Content. Specialized and everyday vocabulary, some 
common abbreviations, functional conversational 
dialogue 

Environment. Familiar, comfortable setting, such as 
the classroom 

Estimated Time To Complete Task. 30 to 45 minutes 
Materials. 

• “George’s Annual Physical” Written Dialogue 

• Vocabulary Matching Answer Sheet (Answer 
Sheet 1) 

• Comprehension Questions (Answer Sheet 2) 

• Pens, pencils, highlighters 

Knowledge and Strategies Assessed at 
Performance Level 4 

Proficient performance requires: 

• Recognizing speciahzed abbreviations and 
vocabulary 

• Familiarity with everyday and some (relatively 
basic) specialized content knowledge and 
vocabulary and with the forms/functions of 
abbreviations 

• Locating important information using a range of 
strategies such as skimming, scanning, identifying 
key words, or using titles and graphics/pictures 



• Monitoring and enhancing comprehension using a 
range of strategies such as posing and answering 
questions, trial and error, or adjusting reading pace 

• Organizing information using some strategies such 
as recall, restatement, simple sequencing and simple 
categorization 

• Recalling prior knowledge and combining it with 
new information 



Additional Knowledge and Strategies That 

May Affect Performance 

• Some familiarity with dialogue structure 

• Familiarity with multiple response formats 
(following directions, vocabulary matching, writing 
short answers) on simple forms 

• Some understanding of the “annual physical exam” 
and common doctor-patient protocols in the United 
States 



Evidence 

Observations. (None in this task) 

Work Products. 

• Responses to the vocabulary word and definition 
matching items (Answer Sheet 1 ) 

• Short written answers to open-ended questions 
(Answer Sheet 2) 
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Model Assessment Task: George’s Annual Physical 
Step-by-Step Procedures for Administration 

This task can be administered to a whole class or to a small group of learners. While there is no time limit for this 
task, it should require no more than 45 minutes to complete. You may repeat any of the steps as needed, but you 
must speak in English and read the steps exactly as they are written in the scripts. You may also model any 
requirement of the task to help clarify, as needed 



Step 1 

Explain the task requirements and expectations: 

“For this task, you will be asked to read a written 
conversation between a doctor and a patient. You will 
show how well you understand the conversation by 
answering some questions about it.” 

Step 2 

Pass out copies of the “George’s Annual Physical” 
written dialogue and Answer Sheet 1. 

Step 3 

Explain the task procedure by holding up the 
dialogue and Answer Sheet 1 in turn while giving 
these instructions: 

“In a moment, I will ask you to begin reading this 
conversation called ‘George’s Annual Physical’ silently to 
yourselves. Before you begin reading, let me explain 
Answer Sheet 1. 

“On Answer Sheet 1, you will match a vocabulary word 
from the reading in the left column with its meaning in 
the right column. You will put the letter of the correct 
meaning (A through I) in the space next to the word. You 
may reread the conversation, and you may make notes 
or highlight on the reading or the answer sheet. You 
can circle or highlight the vocabulary words in the 
conversation. Sometimes seeing the word as it is used in 
conversation can help you find the meaning of the word. 

“Are there any questions before we begin?” 

Step 4 

Tell the learners to begin reading the dialogue and 
completing Answer Sheet 1: 

“Please begin reading the dialogue called ‘George’s 
Annual Physical’ silently to yourself. Answer the questions 
on Answer Sheet 1 as soon as you are ready. Work on 
your own and do not talk to others while you work. Bring 
your finished answer sheet to me when you are done.” 



Step 5 

When the learners are finished with v4nswer Sheet 1, 
pass out Answer Sheet 2 and ask learners to answer the 
questions: 

“Now, using the information that you read in the 
conversation between George and his doctor, answer 
these questions on Answer Sheet 2. You can reread the 
conversation and you may make notes or highlight on the 
reading or the answer sheet. For some questions, there is 
more than one correct answer. 

“Are there any questions before we begin?” 

Step 6 

Tell the learners to complete Answer Sheet 2. 

“Please answer the questions on Answer Sheet 2. Work on 
your own and do not talk to others while you work. Bring 
the written conversation and your finished answer sheet 
to me when you are done.” 

Step 7 

Collect all task materials and record the approximate 
time each learner has taken to complete the task. 
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Scoring Rubric for George’s Annual Physical — EFF Performance Level 4 

Unable 

to Score Beginning Proficient Advanced 





• Familiarity with everyday 
words and specialized 
vocabuiary needs 
improvement; correctiy 
matches five or fewer 
vocabuiary words in the 
diaiogue text to their 
definitions on Answer 
Sheet 1 


• Shows sufficient familiarity 
with everyday words and some 
specialized content and vocabulary 
by correctly matching six out of 
seven vocabulary words in the 
dialogue text to their definitions in 
Answer Sheet 1 


• Shows strong familiarity 
with everyday words and 
specialized content and 
vocabulary by correctly 
matching the seven 
vocabulary words in the 
dialogue text to their 
definitions in Answer 
Sheet 1 




• Locates appropriate 
information for, and 
correctiy answers five or 
fewer of the first seven 
questions on Answer 
Sheet 2 


• Locates appropriate information for 
and correctly answers six out of the 
first seven questions on Answer 
Sheet 2 


• Accurately locates 
information for and 
answers the first seven 
questions on Answer 
Sheet 2 




• Answers to questions 8 
and 9 are inappropriate or 
incomplete (the answers 
include very little to no 
evidence not found in 
the reading), indicating 
difficulty with recalling prior 
knowledge and combining 
it with information from 
the reading 


• Answers questions 8 and 9 
appropriately and completely with 
few minor errors (the answers 
include evidence not found in the 
reading), indicating accurate 
recalling of prior knowledge and 
the ability to combine it with new 
information 


• Answers questions 8 
and 9 appropriately and 
completely, with additional 
details (the answers include 
elaborate, well-described 
evidence not found in 
the reading), indicating 
accurate recalling of prior 
knowledge and the ability 
to combine it with new 
information 




• Has some difficulty 
completing the reading 
task without direction or 
support 


• Completes the reading task with little 
direction or support 


• Completes the reading task 
with ease, needing little or 
no support 
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George’s Annual Physical 



Dialogue Between George and His Doctor 



George is going to the doctor for his yearly checkup with Dr. Brown. Read the 
conversation between George and his doctor. 

[Before the physical] 



Dr. Brown: 


Good morning, George. Today is your annual physical. How are you doing 
these days? 


George: 


Not so well, doctor. I have a few problems. I can’t sleep at night, I have stomach 
pains and I’m gaining weight. 


Dr. Brown: 


Well, let’s examine you and see what’s going on. 



[After the physical] 



Dr. Brown: 


George, I’m really concerned about your health. You have high blood pressure and 
you’ve gained 20 pounds since your last checkup. Let’s talk about what you eat. 
What’s your diet like? 


George: 


Doctor, I love fast food. Yesterday I had a ham and cheese sandwich at 
McDonald’s for breakfast, all meat pizza for lunch at Pizza Hut, and three beef 
tacos at Taco Bell for dinner. 


Dr. Brown: 


Do you exercise? 


George: 


No, I hate to sweat because my clothes get all wet and smell bad. When I get 
home from work, I sit on the couch and watch television until it’s time to go to 
bed. 


Dr. Brown: 


How’s your job going? Do you have a lot of stress on the job? 


George: 


Yes, lately my job’s been really stressful. The company lost a lot of money last 
year, and my boss has been pressuring me to work harder. I can’t sleep at night 
because I worry about losing my job. 
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George’s Annual Physical 



Dialogue Between George and His Doctor, continued 



Dr. Brown: 


Well, George, you need to make some changes in your life. First of all, you need 
to eat healthy food. Eliminate fat in your diet by not eating fried foods and try to 
eat more fruits, vegetables, beans and whole grains. 


George: 


What do you mean by whole grains? 


Dr. Brown: 


Whole grains include whole wheat, brown rice and oatmeal. You should also cut 
out red meat and eat more fish and chicken. 


George: 


Wow, that’s going to be a big change. What else do 1 need to do? 


Dr. Brown: 


You need to exercise at least three times a week. Start out by walking 20 minutes 
three times a week and slowly increase the time. 


George: 


Exercise, that’s going to be difficult. 1 don’t have a lot of time. 


Dr. Brown: 


Instead of watching TV, you can exercise during the evening. 1 know that this 
sounds like a lot of changes, but if you eat right and exercise, you will lose weight, 
your blood pressure will go down and you will be able to handle stress better. 


George: 


I’ll try to follow your advice, doctor. 


Dr. Brown: 


1 think this examination has been good for you. Let’s make an appointment in two 
months and see how you’re doing. 
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George’s Annual Physical 



Answer Sheet 1 



NAME DATE 



Directions 

Look at the vocabulary words numbered 1 through 7 on this Answer Sheet. 

Match the vocabulary words from the reading in the left column with their 
meanings in the right column. Put the letter of the correct meaning (A through I) 
in the space next to the word. 

You may look at the reading while you complete your answer sheet, and you may 
make notes or highlight on the reading or the answer sheet. You can circle or 
highlight the vocabulary words in the conversation. Sometimes seeing the word 
as it is used in conversation can help you find the meaning of the word. 



Vocabulary Words 


Meanings 


1 . annual 


A. water that comes out of your skin when you exercise 


2. TV 


B. brown rice, whole wheat, oatmeal 


3. examination 


C. an opinion that someone gives you to help you 


4. diet 


D. food a person eats 


5. whole orains 


E. stress 


6. advice 


F. checkup 


7. sweat 


G. television 




H. good morning 




I. every year 
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George’s Annual Physical 



Answer Sheet 2 



NAME DATE 



Directions 

Using the information that you read in the conversation between George and his doctor, 
answer these questions. You can reread the conversation and you may make notes or 
highlight on the reading or the answer sheet. For some questions, there is more than one 
correct answer. 



1 . George has some medical problems. List two problems that he tells the doctor about. 



2. What did George eat for lunch yesterday? 



3. Why has George gained 20 pounds? 



4. Does George like to sweat? Why or why not? 



5. Why can’t George sleep at night? 
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George’s Annual Physical 



Answer Sheet 2, continued 



6. List two changes that George can make to help him manage his stress. 



7. Why does George feel that exercise will be difficult for him? 



8. Think about times that you, or other people you know, have been in situations like George’s. 
From what you know about those times, do you think George will follow the doctor’s orders? 
Why or why not? 



9. Think about your own lifestyle. Based on what you read, would you make any changes 
to be healthier? If yes, what changes and why? If no, why not? 
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George’s Annual Physical 



Answer Key for Answer Sheet 1 



NAME DATE 



Directions 


Vocabulary Words 


Meanings 




Look at the vocabulary words 
numbered 1 through 7 on this Answer 


1 


1 . annual 


A. water that comes out of your skin 


Sheet. Match the vocabulary words 
from the reading in the left column 
with their meanings in the right 


G 


2. TV 


when you exercise 
B. brown rice, whole wheat. 


oatmeal 


column. Put the letter of the correct 
meaning (A through I) in the space 


F 


3. examination 


C. an opinion that someone 


gives you to 


next to the word. 

You may look at the reading while you 


D 


4. diet 


help you 

D. food a person eats 




complete your answer sheet, and you 
may make notes or highlight on the 


B 


5. whole grains 


E. stress 




reading or the answer sheet. You can 
circle or highlight the vocabulary 


C 


6. advice 


F. checkup 




words in the conversation. Sometimes 
seeing the word as it is used in 


A 


7. sweat 


G. television 




conversation can help you find the 
meaning of the word. 






H. good morning 





I. every year 



Answer Key for Answer Sheet 2 



NAME 



DATE 



Directions 

Using the information that you read in the conversation between George and his doctor, answer these questions. 
You can reread the conversation and you may make notes or highiight on the reading or the answer sheet. For 
some questions, there is more than one correct answer. 

1 . George has some medical probiems. List two probiems that he teiis the doctor about. 

Any two of these are acceptable answers: 

• He can’t sleep at night. 

• He has stomach pains. 

• He’s gaining weight. 



2. What did George eat for iunch yesterday? 

All meat pizza 
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Answer Key for Answer Sheet 2, continued 



3. Why has George gained 20 pounds? 

Acceptable answers: 

• He eats unhealthy foods or He eats fatty foods or Because of his diet 

• He doesn’t exercise 



4. Does George like to sweat? Why or why not? 

No, because his clothes get wet and smelly 



5. Why can’t George sieep at night? 

Acceptable answers: 

• He worries about losing his job 

• His job is stressful 

6. List two changes that George can make to help him manage his stress. 

Answers will vary; a proficient answer will list two changes that should be related to eating healthy food 
and/or getting more exercise. 



7. Why does George feei that exercise wili be difficuit for him? 

He doesn’t have a lot of time. 



8. Think about times that you, or other people you know, have been in situations like George’s. From what 
you know about those times, do you think George will follow the doctor’s orders? Why or why not? 

Answers will vary; a proficient answer will explain why the learner thinks George will or will not follow the 
doctor’s orders and include information not provided in the reading: 

Example 1 — Yes, George is worried about the problems that he is having. People who are overweight and 
have high blood pressure can get even sicker if they don’t try to get healthier. 

Example 2 — No, once people like George get stuck in the habit of eating poorly and watching TV, it is hard 
for them to stop. 



9. Think about your own lifestyle. Based on what you read, would you make any changes to be healthier? If yes, 
what changes and why? If no, why not? 

Answers will vary; a proficient “yes” answer should explain what changes they would make and why they would 
make them and should be appropriate to the context of being healthier. A proficient “no” answer should explain 
why they would not make any changes and should be appropriate to their health. 
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Design Specifications and Scoring Rubric Templates 
for Accountability Assessments 

PERFORMANCE LEVEL 5: Read With Understanding 



Characteristics of Assessment Tasks 

Action — Performance Goal. Reading to locate, 
summarize, analyze and apply information to make 
a decision or accomplish an action 

Type of Text. Short, dense texts (labels, instructions) 
or longer texts (essays, articles, case studies) that may 
have multiple sections or formats that serve multiple 
purposes (mass transit brochures, Web sites, 
newspapers, medicine labels) 

Content. May include specialized content, 
abbreviations and speciahzed vocabulary 

Environment. One or more familiar or comfortable 
settings 

Estimated Time To Complete Task. One to three 
hours 

Materials. Specihc text(s) as well as tools such as 
dictionaries, written questions or checkhsts to help 
locate important information in text; charts (for 
placement of information); question/answer sheets; 
notes for oral presentations; other documentation 
forms 

Knowledge and Strategies Assessed at 
Performance Level 5 

• Recognizing and interpreting abbreviations and 
speciahzed vocabulary 

• Demonstrating famiharity with everyday and 
some speciahzed content knowledge and vocabulary 
and with paragraph structure and document 
organization 

• Locating important information, reading 
identified sections for detail and determining 
missing information using a wide range of strategies 



• Monitoring and enhancing comprehension using a 
wide range of strategies 

• Organizing and analyzing information and 
reflecting upon its meaning using a range of 
strategies such as classification, categorization, 
and comparison/contrast 

• Evaluating prior knowledge against new 
information in texts to enhance understanding 
of the information 



Evidence 

Observations. Written notes by assessor to document 
adult learner’s completion of task; notes by assessor 
on oral presentations 

Work Products. 

• Oral or written answers to provided questions about 
work process and strategies 

• Information from reading placed on a provided 
chart 

Task Structure 

Tasks may be complex and structured and include 
guides, worksheets, and tools such as a dictionary or 
a computer 
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Scoring Rubric Template — EFF Performance Level 5 

Unable 

to Score Beginning Proficient Advanced 





• Recognizes and uses 
some abbreviations and 
specialized vocabulary, but 
may confuse meanings and 
functions 


• Recognizes and accurately 
interprets abbreviations and 
specialized vocabulary 


• Interprets and uses 
abbreviations and specialized 
vocabulary effectively and 
appropriately 




• Accurately locates and 
reads some key information 


• Accurately locates important 
information, reads identified 
sections for detail and determines 
missing information using a wide 
range of strategies 

• Successfully monitors and 
enhances comprehension using 
a wide range of strategies 


• Provides accurate and 
detailed summaries and 
organizational schemes for 
Information 




• May provide responses 
to the task that are 
incomplete or contain 
major factual or 
organizational errors 


• Effectively organizes and analyzes 
information and reflects upon 
its meaning using a range of 
strategies 






• Draws on prior knowledge 
but may have difficulty 
using new information to 
evaluate It 


• Uses new information to evaluate 
prior knowledge 


• Begins to synthesize 
prior knowledge and new 
information to develop some 
new positions 




• May have difficulty 
completing reading task 
without direction or support 


• Completes reading task with little 
direction or support and without 
major errors 


• Completes reading task with 
little or no support and with 
only a few minor errors 
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Model Assessment Task 

PERFORMANCE LEVEL 5: Read With Understanding 

Using the Web To Find Out How To Write a Resume 



Characteristics of Assessment Tasks 

Action — Performance Goal. Read a scenario about 
a neighbor who wants to write a resume, instructions 
about how to navigate the Internet, and information 
taken from a Web site to find instructions and advice 
for the neighbor on resume writing 

Type of Text. One-page scenario with narrative 
paragraphs and complex sentence structure; one-page 
list of bulleted points on Internet navigation; three 
pages from a Web site (monster.com), two of which 
contain densely written information 

Content. Everyday vocabulary; some specialized 
vocabulary about employment, resumes, the Internet 
and the Web 

Environment. Familiar, comfortable setting, such as 
the classroom 

Estimated Time To Complete Task. 45 minutes 

Materials. 

• Written scenario 

• Instructions for navigating the Internet 

• Three pages from the monster.com Web site 
(Figures 1, 2 and 3) 

• Answer Sheet 1 

• Answer Sheet 2 

• Highlighters, pens/pencils, dictionaries (English 
and bilingual dictionaries) 

Knowledge and Strategies Assessed at 
Performance Level 5 

Proficient performance requires: 

• Recognizing and interpreting abbreviations and 
some speciahzed/technical vocabulary 

• Familiarity with everyday and some specialized 
content knowledge and vocabulary and with 
paragraph structure and document organization 



• Eocating important information, reading identified 
sections for detail and determining missing 
information using a wide range of strategies such as 
skimming; scanning; and using headings/titles, key 
words, context clues and graphics/pictures 

• Monitoring and enhancing comprehension using a 
wide range of strategies such as recalhng, restating 
and summarizing information from multiple 
sources 

• Organizing and analyzing information and 
reflecting upon its meaning using a range of 
strategies such as classification, categorization 
and comparison/contrast 

• Using new information to evaluate prior knowledge 

Additional Knowledge and Strategies That 
May Affect Performance 

• Some familiarity with the Internet, the Web and 
Web site navigation and some very basic familiarity 
with the purpose and content of resumes 

Evidence 

Observations. (None in this task) 

Work Products. 

• Completed question-and-answer forms 
(Answer Sheets 1 and 2) 
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Model Assessment Task: Using the Web To Find Out How To Write a Resume 
Step-by-Step Procedures for Administration 

This task may be administered to an individual or to a group of learners. There is no time limit on this task, but 
it should require no more than 45 minutes to complete. You may repeat any of the steps as needed, but you must 
speak in English and you must read the steps exactly as they are written in the scripts. You also may model any 
requirement of the task to help clarify, as needed. 



Step 1 

Explain the task requirements and expectations: 

“This task assesses how weU you can read and understand 
some instructions for navigating an Internet Web site and 
information about one person who will need your advice 
about writing a resume. You also will be assessed on how 
well you read and understand information about resume 
writing taken from a Web site. You will be asked to 
describe how you would use the Internet to get informa- 
tion about writing a resume and you will be asked to 
show how well you understand the information from the 
Internet by answering questions about what you read.” 

Step 2 

Pass out the scenario, the Web site navigation instruc- 
tions and the Web site page (Figure 1). Ask learners to 
read these materials: 

“Read this scenario about your neighbor, Helen Jones, 
silently to yourself. When you are finished reading the 
story, took at the instructions for navigating a Web site 
and read these instructions silently to yourself 

“After reading these two pages, look at Figure 1. Imagine 
that you go to the Web to find information to help Helen 
with her resume. You decide to look at monster.com, a 
Web site designed to help people find jobs. Figure 1 is 
taken from the monster.com Web site. Read over the 
information on this page. If there are words that you 
do not understand, you may use a dictionary for 
clarification. You can also make notes or highlight on 
any of the materials you receive.” 

Pass out copies of Answer Sheet 1. Ask the learners to 
answer the first two questions on the Answer Sheet 
based on the information they read in the scenario, the 
instructions for navigating the Internet and Figure 1: 

“You will use the information you read from the 
monster.com Web site, from the scenario, from the Web 
site navigation instructions, and also any knowledge you 
may have already about writing resumes or using the 
Internet to answer the questions on your Answer Sheet. 
You may reread any of the materials you have. Are there 
any questions before we begin?” 



Step 3 

After answering any questions, tell learners to begin: 

“Read the documents I gave you and answer the 
questions on Answer Sheet 1. When you have finished, 
please look up.” 

Step 4 

When learners have finished the first part of this task 
and answered the questions on Answer Sheet 1, pass out 
Figures 2 and 3. Ask them to read the information on 
these pages and to answer the two questions on Answer 
Sheet 2: 

“Now imagine you decide to click on two of the links on 
the monster.com Web site that might have information to 
help your friend Helen with her resume. The first link 
brings you to some information about what to do on a 
resume if you’ve been fired and have other negatives in 
your work history. This information can be found on 
Figure 2. The second link brings you to some information 
about having too little experience in the job market. 

This information can be found on Figure 3. Read the 
information provided on both of these sheets. You may 
make notes or highlight on any of these materials. If there 
are words that you do not understand, you may use a 
dictionary for clarification. 

“You will use the information that you read about 
resumes, the information you have in the scenario about 
Helen Jones, and any knowledge you may have already 
about writing resumes or using the Internet to answer 
questions 1 and 2 on Answer Sheet 2. 

“Are there any questions before we begin?” 

Step 5 

After answering any questions, tell learners to begin. 

“Read the documents I gave you and answer the two 
questions on the Answer Sheet.” 

Step 6 

Collect all task materials and record the approximate 
time each learner takes to complete the task. 
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EFF Performance Level 5 

Unable 

to Score Beginning Proficient Advanced 





• Familiarity with everyday 
words and specialized 
content and vocabulary 
needs improvement; 
correctiy identifies one or 
fewer iinks for question 1 
on Answer Sheet 1 that are 
relevant to Helen Jones’ 
situation 


• Shows sufficient familiarity 
with everyday words and some 
specialized content and vocabulary 
by correctly identifying two links 
for question 1 on Answer Sheet 1 
that are relevant to Helen Jones’ 
situation 


• Shows strong familiarity 
with everyday words and 
specialized content and 
vocabulary by correctly 
identifying three links for 
question 1 on Answer 
Sheet 1 that are relevant 
to Helen Jones’ situation 




• Demonstrates a limited 
ability to locate information 
by being able to identify 
one or fewer relevant links 
in answer to question 1 and 
having difficulty explaining 
reasoning for choice of link 
in question 2 on Answer 
Sheet 1 


• Demonstrates sufficient ability 
to locate information by correctly 
identifying two relevant links and 
by giving adequate reasoning for 
choice of link in question 2 on 
Answer Sheet 1 


• Demonstrates a very good 
ability to locate information 
by identifying three relevant 
links in answer to question 
1 and by providing a 
detailed, well-thought-out 
reasoning for choice of link 
in question 2 on Answer 
Sheet 1 




• Responses to questions 1 
and 2 on Answer Sheet 2 
could be improved; does 
not indicate the effective 
application of prior 
knowledge; little to no 
application of new 
knowledge from Figures 2 
and 3 


• Responses to questions 1 and 2 
on Answer Sheet 2 reveal some 
ability to recall prior knowledge and 
combine it with new information to 
appropriately answer questions; 
uses new information from Figures 2 
and 3 appropriately with few major 
errors 


• Strong, effective use of 
prior knowledge to answer 
questions 1 and 2 on 
Answer Sheet 2; uses new 
information from Figures 2 
and 3 to provide a 
reasonable explanation for 
answers to questions 1 and 
2 with few minor errors 




• Has some difficulty 
completing the reading task 
without direction or support 


• Completes reading task with little 
apparent difficulty, needing little 
direction or support 


• Completes reading task 
with ease, needing little or 
no support 
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Scenario for Helping a Neighbor Find a Job 



Your neighbor, Helen Jones, is trying to find a job and would like to 
write a resume. She has had some problems in her past job and is 
worried about writing her resume. She has heard about Web sites on 
the Internet that help people write resumes but she doesn’t have 
much experience with computers. She has come to you for help. 

Helen used to work as an assistant manager at a grocery store. 
Before that, she worked as a cashier in the same store. She was 
promoted to assistant manager three months after she had started 
working as a cashier because she was hardworking, very well 
organized and respected by her co-workers. She was given an 
“employee of the month” award after her fourth month at work. 

Six months after Helen was promoted, she broke her leg and was 
in the hospital for two weeks. During that time, she could not go to 
work at all. When she was able to return to work, she could only 
work limited hours and had problems standing for many hours, 
which the job demanded. She also had to go to many doctor’s 
appointments, often during her shift. The manager of the store fired 
her after she had missed work several times without explanation. 

Helen is 23 years old. The cashier job was the first job that she ever 
had. After being fired, she got her GED and is interested in learning 
more about how to use computers. Her leg is healed and she wants 
to get another job as a cashier, but she is concerned that her lack of 
experience and having been fired will make getting a new job 
difficult. 

You can help Helen with her resume by using information from the 
Internet. 
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Instructions for Navigating a Web Site 



The Internet is a large collection of computers connected together. 
Computers “talk” to each other and can send and receive information 
like electronic mail, picture files and music files. These computers are 
commonly referred to as the World Wide Web, or sometimes just the 
Web or the ’net. Individuals can use the Internet to get information on 
almost any subject as well, which is why some have called the World 
Wide Web the “information superhighway.” 



Here are some basic instructions for how to look at a Web site on the 
Internet: 

You move to a new page by typing its URL — its location (address) on 
the Web. URLs normally begin with http://, followed by one or more 
names that identify the address. For instance, http://netscape.com. 

1 . Click the Location Bar to select the URL that is already there. 

2. Type the URL of the page you want to visit. The URL you type 
replaces any text already in the Location Bar. 

3. Press Enter. 



Clicking a Link: 

Most Web pages contain links you can click on to move to other pages. 

1. Move the pointer until it changes to a pointing finger. This happens 
whenever the pointer is over a link. Most links are underlined text, 
but buttons and pictures can also be links. 

2. Click the link once. While the network locates the page that the link 
points to, status messages appear at the bottom of the window. 
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Answer Sheet 1 



NAME DATE 



Directions 

Read the scenario and the instructions for navigating a Web site silently to yourself. 
When you have finished reading, look at Figure 1 . Imagine that you go to the World 
Wide Web to find information to help Helen with her resume. You decide to look at 
monster.com, a Web site designed to help people find jobs. Figure 1 is taken from 
the monster.com Web site. Read over the information on this page. You may make 
notes or highlight on any of these materials. If there are words that you do not 
understand, you may use a dictionary for clarification. When you have finished 
reading these figures, answer the questions on this page. 

Using the information you read from the monster.com Web site (Figure 1), from the 
scenario, from the Web site navigation instructions, and also any knowledge you 
may already have about writing resumes or using the Internet, answer questions 1 
and 2. You may reread any of the materials you have. 



1 . Identify three links from Figure 1 that you would click on to find information that 
could help Helen Jones. 



2. Choose one of the three links that you listed above and explain why you think 
this link will help Helen write her resume. 
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Answer Sheet 2 



NAME DATE 



Directions 

Imagine you decide to click on two of the links on the monster.com Web site that 
might have information to help your friend Helen with her resume. The first link 
brings you to some information about what to do on a resume if you’ve been fired 
or have other negatives in your work history. This information can be found on 
Figure 2. The second link brings you to some information about having too little 
experience in the job market. This information can be found on Figure 3. Read the 
information provided on both of these sheets. You may make notes or highlight on 
any of these materials. If there are words that you do not understand, you may use 
a dictionary for clarification. When you have finished reading these figures, answer 
the questions on this page. 

You will use the information that you read in these figures, the information you have 
in the scenario about Helen Jones, and any knowledge you may have already about 
writing resumes or using the Internet to answer these two questions. 

1. After reading Figure 2, what advice would you give Helen about writing her 
resume? 



2. After reading Figure 3, what advice would you give Helen about writing her 
resume? 
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Figure 1 
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Answer Key for Answer Sheet 1 



NAME DATE 



Directions 

Read the scenario and the instructions for navigating a Web site silently to yourself. 
When you have finished reading, look at Figure 1 . Imagine that you go to the World 
Wide Web to find information to help Helen with her resume. You decide to look at 
monster.com, a Web site designed to help people find jobs. Figure 1 is taken from 
the monster.com Web site. Read over the information on this page. You may make 
notes or highlight on any of these materials. If there are words that you do not 
understand, you may use a dictionary for clarification. When you have finished 
reading these figures, answer the questions on this page. 

Using the information you read from the monster.com Web site (Figure 1), from the 
scenario, from the Web site navigation instructions, and also any knowledge you 
may already have about writing resumes or using the Internet, answer questions 1 
and 2. You may reread any of the materials you have. 



1 . Identify three links from Figure 1 that you would click on to find information that 
could help Helen Jones. 

Any three of the following are acceptable answers: 

• Resume Dilemma: Being Fired 

• Resume Dilemma: Recent Graduate 

• Resume Dilemma: Too Little Experience 

• Tips for Creating a Concise Resume 



2. Choose one of the three links that you listed above and explain why you think 
this link will help Helen write her resume. 

Answers will vary; a proficient answer will be appropriate to the link chosen and 
to helping Helen write her resume. 
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Answer Key for Answer Sheet 2 



NAME DATE 



Directions 

Imagine you decide to click on two of the links on the monster.com Web site that 
might have information to help your friend Helen with her resume. The first link 
brings you to some information about what to do on a resume if you’ve been fired 
and have other negatives in your work history. This information can be found on 
Figure 2. The second link brings you to some information about having too little 
experience in the job market. This information can be found on Figure 3. Read the 
information provided on both of these sheets. You may make notes or highlight on 
any of these materials. If there are words that you do not understand, you may use a 
dictionary for clarification. When you have finished reading these figures, answer the 
questions on this page. 

You will use the information that you read in these figures, the information you have 
in the scenario about Helen Jones, and any knowledge you may have already about 
writing resumes or using the Internet to answer these two questions. 



1. After reading Figure 2, what advice would you give Helen about writing her 
resume? 

Answers will vary; a proficient answer will be appropriate to helping Helen write 
her resume and include information drawn from Figure 2 and relevant prior 
knowledge about writing resumes or using the Internet. 

2. After reading Figure 3, what advice would you give Helen about writing her 
resume? 

Answers will vary; a proficient answer will be appropriate to helping Helen write 
her resume and will include information drawn from Figure 3 and relevant prior 
knowledge about writing resumes or using the Internet. 
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Design Specifications and Scoring Rubric Templates 
for Accountability Assessments 

PERFORMANCE LEVEL 6: Read With Understanding 



Characteristics of Assessment Tasks 

Action — Performance Goal. Reading to locate, 
summarize, analyze and apply information to make a 
decision or accomplish an action 

Type of Text. Long, complex, authentic texts 
(instructional manuals, bulletins) organized in 
multiple sections or formats that serve multiple 
purposes (cell phone manual, financial aid bulletin) 

Content. Includes specialized content and technical 
vocabulary 

Environment. One or more familiar or comfortable 
settings; setting may feature less familiar, task-related 
elements (cell phones, computers) 

Estimated Time To Complete Task. One to five 
hours 

Materials. Specific text(s) as well as other necessary 
tools, documentation forms or question/answer 
sheets 

Knowledge and Strategies Assessed at 
Performance Level 6 

• Recognizing and interpreting terms, signs, symbols, 
acronyms and abbreviations 

• Demonstrating familiarity with extensive specialized 
content knowledge and vocabulary and with the 
organization of long, complex prose and complex 
documents 

• Using a wide range of strategies to guide reading of 
long texts 

• Identifying both directly stated and implied 
important information 

• Monitoring and enhancing comprehension using a 
wide range of strategies such as brainstorming and 
question formulation techniques 



• Organizing and analyzing information and 
reflecting upon its meaning using a wide range of 
strategies such as applying relevant information to 
multiple scenarios, summarizing, and drawing “big 
picture” conclusions and generalizations from 
detailed reading 

• Integrating prior knowledge with new information 
in texts to develop deep understanding of the 
information 



Evidence 

Observations. 

• Oral interviews with adult learners, documented 
by assessor, about work process/strategies 

• Written observations by assessor to document 
completion of task 

Work Products. 

• Written answers in sentences or short paragraphs to 
provided questions about work process or strategies 

Task Structure 

Tasks may be complex, structured or unstructured, 

and include suggested reading strategies, worksheets, 

and use of tools such as a dictionary or a computer 
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Scoring Rubric Template — EFF Performance Level 6 

Unable 

to Score Beginning Proficient Advanced 





• Recognizes and uses 
some technical terms, 
signs, symbols, acronyms 
and abbreviations, but may 
confuse meanings and 
functions 


• Recognizes and accurately 
interprets technical terms, 
signs, symbols, acronyms and 
abbreviations 


• Interprets and uses technical 
terms, signs, symbols, 
acronyms and abbreviations 
effectively and appropriately 




• Synthesizes or summarizes 
key information from long, 
complex text(s) with some 
minor inaccuracies 


• Accurately synthesizes, 
summarizes or draws conclusions 
from the information presented 
in long, complex text(s) 


• Accurately and 
comprehensively synthesizes, 
summarizes or draws 
conclusions from information 
in long, complex text(s) 




• Uses a wide range of 
strategies to enhance 
comprehension 


• Uses a wide range of strategies 
and overcomes some barriers to 
comprehension 


• Uses a wide range of 
strategies and easily 
overcomes barriers to 
comprehension 




• Synthesizes and begins to 
integrate prior knowledge 
and new information to 
enhance understanding 
and ability to apply 
information 


• Effectively integrates new 
information with prior knowledge 
to enhance understanding and 
ability to apply information 


• Easily Integrates new 
Information with prior 
knowledge to enhance 
understanding and ability 
to apply Information 




• Completes reading 
task without direction 
or support, but some 
responses may be 
incomplete or contain 
major factual errors 


• Completes reading task with little 
direction or support and without 
major errors 


• Completes reading task with 
little or no support and with 
only a few minor errors 
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Model Assessment Task 

PERFORMANCE LEVEL 6: Read With Understanding 

Dealing With Bullying 



Characteristics of Assessment Tasks 

Action — Performance Goal. Read an article on 
bullying to offer information to a parent about 
identifying and dealing with buUying behavior in 
a child 

Type of Text. Eight-page, multisection excerpt from 
an Internet article on bullying, Bullying: Information 
for Parents and Teachers (includes citations) 

Content. General and some specialized vocabulary 
and content on a social issue designed for an audience 
(such as parents) with an interest in the issue 

Environment. Familiar, comfortable setting, such as 
the classroom 

Estimated Time To Complete Task. 1 .5 hours 
Materials. 

• “Bullying: Information for Parents and Teachers” 

• Question and Answer Sheet (Answer Sheet I) 

• Pens, pencils, highlighters, dictionaries 

Knowledge and Strategies Assessed at 
Performance Level 6 

Proficient performance requires: 

• Recognizing and interpreting terms, signs, symbols, 
acronyms and abbreviations such as terms used in 
technical manuals, complex application forms or 
other technical literature 

• Familiarity with extensive speciahzed content 
knowledge and vocabulary and with the 
organization of long, complex prose (manuals, 
bulletins) and complex documents (forms, 
schedules, charts) 



• Using a wide range of strategies (drawing on prior 
experience with a variety of prose and document 
formats, previewing, predicting and using text 
organizers such as index, table of contents, headings, 
bold print, bullets) to guide reading of long texts 

• Identifying both directly stated and implied 
important information 

• Monitoring and enhancing comprehension using a 
wide range of strategies such as brainstorming and 
question formulation techniques 

• Analyzing information and reflecting upon its 
meaning using a wide range of strategies such 
as applying relevant information to multiple 
scenarios, summarizing, and drawing “big picture” 
conclusions and generahzations from detailed 
reading 

• Integrating new information with prior knowledge 
to develop deep understanding and make decisions/ 
prepare to act 

Additional Knowledge and Strategies That 
May Affect Performance 

• General familiarity with child development and 
behavior 

• Familiarity with required response format 

Evidence 

Observations. (None in this task) 

Work Products. 

• Written responses on Answer Sheet 1 
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Model Assessment Task: Dealing With Bullying 



Step-by-Step Procedures for Administration 

This task can be administered to individual students or small groups. While there is no time limit to complete 
the task, it should require no more than 1.5 hours to complete. You may repeat any of the steps as needed, but 
you must speak in English and read the steps exactly as they are written in the script. You also may model any 
requirement of the task to help clarify, as needed. 



Step 1 

Explain the task requirements and expectations and 
hand out the reading passage and Answer Sheet 1: 

“This is an assessment of your ability to read with 
understanding. I have given you a copy of an excerpt 
from an article on the subject of bullying. In a moment, I 
will ask you to begin reading this article. After you have 
finished reading the article, you will write definitions and 
answer questions on the answer sheet. The questions ask 
you to find information in the article and, in some cases, 
to analyze and interpret that information in light of your 
own knowledge and experience. When you are writing 
answers to questions, you may look back at the article to 
find the information you need to respond to the 
questions.” 



Step 3 

Observe the students as they read the article and make a 
note of how long each spends in the initial reading of 
the article and how long it takes to completely finish 
writing answers to the questions on the answer sheet. 

Step 4 

When all learners have finished writing, make sure that 
all materials, including the article and answer sheets, 
have been collected. 



Step 2 

Tell the students to begin working on the task: 

“Begin reading the article now. You may use a dictionary 
if you come across a word you do not understand. When 
you have finished reading once through the entire article, 
take out the answer sheet, read the directions and begin 
writing your answers. In completing the answer sheet, 
you should look back at the article to find relevant 
information, but do not simply copy the words of the 
article or definitions from the dictionary in your written 
answers. Use your own words as much as possible. You 
may also use knowledge that you already have about this 
subject in your responses. In some cases, there is more 
than one right answer to a question. When you have 
completed writing answers to all the questions, please 
hand in the article and your answer sheet.” 
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Unable 

to Score Beginning Proficient Advanced 





• Defines fewer than six 
vocabuiary items correctiy 
and circies one or more 
incorrect responses to 
questions 2a and 2b 


• Correctly defines six or more 
vocabulary items and correctly 
identifies answers to questions 
2a and 2b 


• Correctly defines all 
vocabulary items and 
effectively uses specialized 
vocabulary in written 
responses to questions 2a 
and 2b 




• Has difficulty locating 
information needed to 
respond to questions 


• Locates relevant information in 
the text and uses it in answering 
questions 2, 3 and 4 


• Quickly and easily locates 
relevant information in the 
text and uses it in answering 
questions 2, 3 and 4 




• Summarizes and 
synthesizes main ideas 
with minor inaccuracies 
in answers to questions 
3 and 4 


• Accurately summarizes and syn- 
thesizes main ideas in answers to 
questions 3 and 4 


• Accurately and 
comprehensively summarizes 
and synthesizes main ideas 
in answers to questions 
3 and 4 




• Written responses to 
questions 3 and 4 are 
incomplete or inaccurate 


• Written responses to questions 3 
and 4 are complete and accurate 


• Written responses to 
questions 3 and 4 are 
complete, concise and 
accurate 




• Responses to questions 3 
and 4 demonstrate some 
difficulty with integrating 
new information with prior 
knowledge and experience 
to develop a deeper 
understanding of the text 
and be able to make 
informed decisions 


• Responses to questions 3 
and 4 demonstrate a thorough 
understanding of the text and the 
ability to make informed decisions 
by integrating new information 
with prior knowledge and 
experience 


• Responses to questions 3 
and 4 demonstrate an ability 
to integrate new information 
easily and comprehensively 
with prior knowledge and 
experience to gain a deeper 
understanding of the text and 
make informed decisions 
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Bullying: Information for Parents and Teachers 

Marlies Sudermann, Ph.D., C. Psych. 

Peter G. Jaffe, Ph.D., C.Psych. 

Elaine Schieck, B.A. 

19 9 6 



What Is Bullying? 

Bullying can be defined as repeated and systematic 
harassment and attacks on others. Bullying can be perpe- 
trated by individuals or groups. Bullying takes many 
forms, and can include many different behaviors, such as: 

• physical violence and attacks 

• verbal taunts, name-calling and put-downs 

• threats and intimidation 

• extortion or stealing of money and possessions 

• exclusion from the peer group 

Racially or ethnically-based verbal abuse and gender- 
based put-downs are also found in the bullying situation. 

How prevalent is bullying? 

Studies in several countries have consistently shown that 
at least 15 percent of students in schools are involved. 

• About 9 percent are victims. 

• About 7 percent buUy others repeatedly. 

• More students in younger grades are victimized. 

• Boys are more likely to be bullies than girls. 

Dynamics of Bully-Victim Situations 

• A power differential exists between the bully and the 
victim. 

• Bullies tend to be confident, aggressive, and lack empa- 
thy for the victim. 

• Bullies come from homes where there is poor supervi- 



sion, and modeling of and tolerance for aggressive 
behavior. 

• Victims tend to be quiet, passive children with few 
friends. 

• Victims do not respond effectively to aggressive 
actions. 

• BuUying is often done so that adults are not aware of it. 

• Victims are ashamed, and often don't tell an adult. 

Research on Bullying 

Professor Dan Olweus, the pre-eminent researcher of 
bullying among school-age children and youth, defines 
buUying as follows: 

A student is being bullied or victimized when he or 
she is exposed, repeatedly and over time, to nega- 
tive actions on the part of one or more other stu- 
dents (Olweus, 1993, p. 9). 

A power imbalance is found at the heart of the bullying 
dynamic. A student who is stronger, more aggressive, 
bolder, and more confident than average typically bullies 
other students who are weaker, more timid, and who 
tend not to retaliate or act in an assertive manner. 
Sometimes older students buUy younger ones, or upper 
year students bully new students. Sometimes bullies pick 
on students who are disadvantaged by being new immi- 
grants or are from a cultural minority group. 

As with other interpersonal violence, such as dating vio- 
lence, racial harassment, child abuse, and wife assault, the 
power imbalance is a main factor in understanding what 
is going on. Interventions must take this into account. 
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It is a myth that bullies are insecure underneath their 
bravado. Research indicates that their self esteem is, on 
average, average or above average. It appears that bullies 
tend to come from homes where aggressive strategies to 
conflict resolution is modeled, although more research 
needs to be done on this connection. 

Victims tend to be timid and, in the case of boys, tend to 
be physically weaker and less skilled than bullies. Victims 
tend to lack assertive responses to peer aggression, and 
they tend to be low on skills for making friends. They 
generally do not retaliate when they are picked on, so 
that they come to be seen as “safe” targets for bullying. 

Prevalence of Bullying 

Bullying is an old and widespread problem. Most of us 
can recall episodes of bullying that we or our classmates 
were subjected to during our school years. Research esti- 
mates indicate that the problem affects far more students 
than teachers or parents are aware of A recent study of 
1041 students in four Toronto area schools (Grades K-8) 
showed that the proportion of children who reported 
being victimized more than once or twice over the term 
was between 12 and 15 percent (Pepler, Craig, Ziegler & 
Charach, 1994). The proportion of students who report- 
ed having bullied others more than once or twice over 
the term ranged from 7 to 9 percent. 

Olweus, in his extensive studies over the past twenty 
years in Norway, has found that about 15 percent, or one 
in seven students, are involved in bully/victim problems. 
Of these, about 9 percent are victims, and 7 percent buUy 
others with some regularity (Olweus, 1993). When Dr. 
Olweus and his colleagues looked at very serious bully- 
victim problems, they found that slightly more than 3 
percent of their very large sample were bullied once a 
week or more, while just less than 2 percent of students 
bullied other students that frequently 

Studies from a number of other countries have con- 
firmed that rates of bullying are the same or higher in 
England, the United States, Japan, Ireland, Australia, and 
the Netherlands, among other countries. 



Another important finding from these research studies is 
that most students who are bullied either do not report 
the bullying to adults, or they wait a very long time 
before doing so. The reasons include feelings of shame, 
fear of retaliation for reporting, and fear that adults can- 
not or will not protect the victim in the settings where 
bullying usually takes place: the playground, the hallway 
of the school, or on the way to and from school. 

Olweus and colleagues have found that the percentage of 
students who report being victims of bullying decreases 
with age, over grades 2 to 9. In their sample of over 83,000 
students in Norway they found that while between 16 
and 17 percent of students in Grade 2 reported being bul- 
lied, by Grade 9, the percentages decreased to 3 percent of 
girls and 6.5 percent of boys (Olweus, 1993). A large pro- 
portion of the bullied children in the lower grades report- 
ed being bullied by older children. This again underKnes 
the role of power differentials in bullying. 

In Ganada, the studies on bullying have been fewer and 
much smaller scale than those in Norway so there is less 
information available with regard to patterns over differ- 
ent grade levels. The study by Pepler et al. (1994) does 
not provide a grade breakdown in incidence of bullying. 
Another study which was done by Ryan, Mathews and 
Banner (1993), provides information about aggression 
and victimization in a sample of 457 grade 7, 8, and 9 
students. Ryan et al. (1993) used a substantially different 
set of questions compared to the Olweus and the Pepler 
et al. studies. Therefore, it is not possible to make direct 
comparisons between these studies, as to the rates of bul- 
lying at different grade levels. The students were asked to 
report whether each type of incident had ever happened 
to them, with no time period specified. Therefore, stu- 
dents in each grade may have been reporting incidents 
that happened to them in earlier grades. Also, the stu- 
dents in this study were asked to report on one-time 
occurrences of violence, rather than on repeated patterns 
or bullying. The Ryan et al. study found fairly high rates 
of occurrence of violent incidents, with a wide range of 
violent incidents being reported, from being threatened, 
to harassed, to having lunch money taken, to being 
threatened with a weapon. 
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Gender Differences in Bullying 

Patterns of bullying and victimization are very different 
for boys and girls. Boys are much more likely to report 
being bullies, and perpetrating violent acts on others 
than are girls, at each age. Girls are somewhat less likely 
than boys to be the victims of bullying, although the 
rates are not as discrepant as the bullying (perpetrator) 
rates. 

This suggests that it is important to study whether boys 
victimize other boys, or both boys and girls, and vice 
versa. Olweus (1993) reports that one of his studies, 
conducted with students in grades five to seven, found 
that 60 percent of girls who were bullied were bullied 
only by boys, while another 15-20 percent were bullied 
by both boys and girls. The great majority of boys who 
were bullied (80 percent) were bullied only by boys. 
This shows that it is boys who are more likely to be the 
perpetrators of what Olweus calls “direct” bullying, that 
is, bullying which involves direct physical or verbal 
attacks. He has concluded that girls are more likely to 
use indirect, subtle, social means to harass other girls. 
He refers to behavior such as social exclusion, manipu- 
lation of friendship relationships, spreading rumors, 
etc. However, there appear to be few questions in his 
questionnaires to address this issue. The one question 
he did include was “How often does it happen that 
other students don’t want to spend recess with you and 
you end up being alone?” The results indicated that 
boys and girls were equally likely to have this problem. 
Olweus sees this as a measure of “in direct” bullying, but 
an alternative explanation is that this question may 
address socially rejected children who are not liked, but 
who are not intentionally bullied either. These may sim- 
ply be the less popular children. 

One conclusion about gender differences is that boys 
are more likely to be both the perpetrators and the vic- 
tims of aggressive physical and verbal bullying by peers. 
Another conclusion is that girls are much more often a 
target of bullying by boys than vice versa. Taken togeth- 
er, these conclusions indicate that interventions should 
take into account the higher rates of aggressive behavior 
by males. A third conclusion is that more study is need- 



ed of “indirect” or subtle bullying and of social exclu- 
sion, by both girls and boys. 

What Causes Bullying? 

A number of different factors have been identified which 
contribute to bullying problems. Family, individual, and 
school factors all contribute. 

Family factors: A number of child-rearing styles have 
been found to predict whether children will grow up to 
be aggressive bullies. A lack of attention and warmth 
toward the child, together with modeling of aggressive 
behavior at home, and poor supervision of the child, 
provide the perfect opportunity for aggressive and bully- 
ing behavior to occur (Loeber & Stouthamer-Loeber, 
1986; Patterson, DeBaryshe & Ramsey, 1989; and 
Olweus, 1993). Modeling of aggressive behavior may 
include use of physical and verbal aggression toward the 
child by parents, or use of physical and verbal aggression 
by parents toward each other. The connection between 
witnessing wife assault by children, particularly male 
children, and bully behavior by children toward peers, 
has not been well studied, but studies do indicate that 
aggressive behavior of all kinds is elevated in children 
who witness violence by their father toward their mother 
(Jaffe, Wolfe & Wilson, 1990). 

Individual factors: The best-documented individual 
child factor in bullying is temperament. Temperament 
refers to basic tendencies by children to develop certain 
personality styles and interpersonal behaviors. Children 
who are active and impulsive in temperament may be 
more inclined to develop into bullies. With boys, physical 
strength compared to age peers also seems to be a charac- 
teristic which is associated with bullying, although of 
course there are many strong, physically adept boys who 
never bully. 

School factors: The social context and supervision at 
school have been shown to play a major part in the fre- 
quency and severity of bullying problems. While teachers 
and administrators do not have control over individual 
and family factors which produce children who are 
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inclined to bully, bullying problems can be greatly 
reduced in severity by appropriate supervision, interven- 
tion and climate in a school. 

Supervision of children has been found to be of prime 
importance. Just as low levels of supervision in the home 
are associated with the development of bully problems in 
individual children, so too, are low levels of supervision 
at school, particularly on the playground or schoolyard 
and in the hallways. Also, the appropriateness of inter- 
ventions by adults when they see bullying, or are made 
aware of it are very important. 

The social climate in the school needs to be one where 
there is warmth and acceptance of aU students, and one 
where there are high standards for student and teacher 
behavior toward one another. Teacher attitudes toward 
aggression, and skills with regard to supervision and 
intervention, partly determine how teachers will react to 
bullying situations. Curricula and administrative policies 
and support are also very important. These are further 
outlined in the section on Program that Work. 

Who Becomes a Victim? 

Children who become repeated victims of aggression, 
and bullying, tend to be quiet and shy in temperament. 
They tend not to retaliate or make any assertive respons- 
es to the initial aggression, which is then repeated by the 
bully. Children who become victims typically lack friends 
and social support at school, and they are often not con- 
fident in their physical abilities and strength. 

While most victims do not do anything to provoke the vic- 
timization, there is a subgroup of victims who tend to 
show irritating and inappropriate social behavior. These 
children tend to be impulsive and have poor social skills. 
These “provocative victims” may also try to bully other 
children, so they are both bully and victim (Olweus, 1993). 

What are the Long-term Consequences 
for Victims, Bullies, and Bystanders? 

Victims of bullying typically are very unhappy children 



who suffer from fear, anxiety, and low self-esteem as a 
result of the bullying. They may try to avoid school, and 
to avoid social interaction, in an effort to escape the bul- 
lying. Some victims of bullying are so distressed that they 
commit, or attempt to commit suicide. Several instances 
of suicide by boys who had been severely bullied 
occurred in Norway in the early 1980’s. These tragic 
events mobilized that country to begin a nation-wide 
anti-buUying program (Olweus, 1993). 

Even when bullying does not drive victims to the 
extremes of suicide, victims experience significant psy- 
chological harm which interferes with their social and 
academic and emotional development. The sooner the 
bullying is stopped, the better for the long-term outcome 
for victims. If bullying patterns are allowed to continue 
unchecked, there are long-term consequences for the vic- 
tim. A follow-up study by Olweus (1993b) found that by 
the time former male victims of bullying were in their 
early twenties, they had generally made a positive social 
adjustment, as they had more freedom to choose their 
social and work milieu. However, they were more likely 
to be depressed, and had lower self-esteem than a com- 
parison group who had not been bullied. 

The serious long-term outcomes for bullies are also 
important to recognize. Bullies tend to become aggres- 
sive adults who stand a much higher chance than average 
of obtaining multiple criminal convictions (Olweus, 
1979). These findings by Olweus and his group fit well 
with other studies which have found exactly the same 
outcome for children, especially males, who are aggres- 
sive as children (e.g. Robins, 1978; Loeber & Dishion, 
1983). 

Another important but often overlooked group of chil- 
dren who are affected by bullying are those children who 
are neither victims nor perpetrators of bullying, but who 
see bullying happen to their peers. There are also chil- 
dren who will not take the initiative to buUy themselves, 
but will follow a bully's lead in helping to harass or vic- 
timize a particular child in their class or school. All chil- 
dren, including bystanders, are negatively affected when 
bullying occurs. The bullying may cause anxiety or fear 
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in bystanders. The learning environment is poisoned by 
buUying, particularly when there are no effective inter- 
ventions in the bullying situation. Children who observe 
violent behavior and see that it has no negative conse- 
quences for the bully, will be more likely to use aggres- 
sion in the future. 



What Can Be Done to Stop 
Bullying? 

There are many effective strategies for both teachers and 
parents who wish to stop bullying. An important starting 
point is to realize that much bullying occurs without the 



What Can Parents Do If Their 
Children Are Being Bullied? 

I Ask the child directly. Often 
children do not wish to tell 
their parents due to shame and 
embarrassment, or fear that bullies 
will retaliate if they tell. Look for 
signs such as: fear of going to 
school, lack of friends, missing 
belongings and torn clothing, and 
increased tearfulness and anxiety. 

2 Work with the school immedi- 
ately to make sure your child is 
safe, that effective consequences 
are applied toward the bully, 
and that monitoring at school is 
adequate. Advocate for involvement 
of the bully’s parents. If the bullying 
is happening on the way to and 
from school, arrange for the child to 
get to school with older, supportive 
children, or take him or her until 
other interventions can take place. 

If your child is timid, and 
lacks friends, try to arrange for 
your child to participate in positive 
social groups which meet his or her 
interests. Developing your child’s 
special skills and confidence in the 
context of a positive social group 
can be very helpful. 

Suggest that the school 
implement a comprehensive 
anti-bullying program. A home- 
and-school association meeting 
to discuss and support such an 
initiative can be helpful. 



What To Do If Your Child Is 
Aggressive Or Bullies Others? 

Take the problem seriously. Children 
and youth who bully others often 
get into serious trouble in later life, 
and may receive criminal convic- 
tions. They may have continuing 
trouble in their relationships with 
others. Here are some things you 
can do to turn the situation around. 

I Talk to your child, talk to his or 
her teachers and administrators. 
Keep in mind that a bully will try 
to deny or minimize his or her 
wrong-doing. 

2 Make it clear to your child that 
you will not tolerate this kind of 
behavior, and discuss with your 
child the negative Impact bullying 
has on the victims. Do not accept 
explanations that “it was all In fun.” 

3 Arrange for an effective, non- 
violent consequence, which Is in 
proportion with the severity of your 
child’s actions, and his or her age 
and stage of development. Corporal 
punishment carries the message 
that “might is right.” 

4 Increase your supervision of 
your child’s activities and 
whereabouts, and who they are 
associating with. Spend time with 
your child, and set reasonable rules 
for their activities and curfews. 



5 Co-operate with the school 
in modifying your child’s 
aggressive behavior. Frequent 
communication with teachers 
and/or administrators is important 
to find out how your child is doing 
in changing his or her behavior. 

Praise the efforts your child 
makes toward non-violent and 
responsible behavior, as well as for 
following home and school rules. 
Keep praising any efforts the child 
makes. 

7 If your child is viewing violent 
television shows, including 
cartoons, and is playing violent 
video games, this will increase 
violent and aggressive behavior. 
Change family and child’s viewing 
and play patterns to non-violent 
ones. 

8 Make sure that your child is 
not seeing violence between 
members of his or her family. 
Modeling of aggressive behavior at 
home can lead to violence by the 
child against others at school and 
in later life. 

9 Seek help from a school 

psychologist, social worker, or 
children’s mental health center in 
the community if you would like 
support in working with your child. 
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knowledge of teachers and parents, and that many vic- 
tims are very reluctant to tell adults of their problems 
with bullying. They may be ashamed to be a victim, and 
they are afraid that adults cannot or will not help to 
resolve the situation. They may have been threatened 
with retaliation if they tell. 

Also, adults must re-examine some of their own beliefs 
with regard to interpersonal behavior before they can 
intervene effectively. Many teachers and parents tell chil- 
dren not to “tattle,” and to resolve their problems them- 
selves. In the bullying situation, though, there is a power 
imbalance of some kind which ensures that the victim 
always gets the worst of the interaction. The victim and 
bully both need intervention in order to stop the pattern. 

Some important strategies in stopping bullying are: pro- 
viding good supervision for children; providing effective 
consequences to bullies; using good communication 
between teachers and parents; providing all children 
opportunities to develop good interpersonal skills; and 
creating a social context which is supportive and inclu- 
sive, in which aggressive, bully behavior is not tolerated 
by the majority. 
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Directions 

After you have finished reading the excerpt from the article “Bullying: 
Information for Parents and Teachers,” define the terms in question 1 
and write answers to questions 2, 3 and 4. You may look back at the 
text to find information to answer the questions, but do not copy from 
the text or from the dictionary in writing your response. Use your own 
words. You also may use knowledge that you already have about this 
subject in your responses. In some cases, there is more than one right 
answer. 



1 . Write a short definition or explanation of the following terms. Write answers in your own 
words. Do not just copy from the text or the dictionary. 

a. aggressive behavior 



b. retaliation 



c. perpetrator 



d. victimize 



e. bystander 



f. timid 



g. “indirect” bullying 



EFF ASSESSMENT CONSORTIUM 



103 





Improving Performance, Reporting Results 



Section 2: The Accountability Assessment Prototype for Read With Understanding 



Dealing With Bullying 



Answer Sheet 1, continued 



2. Circle the correct answer(s) to the following questions: 

a. In which of the following country or countries did Dr. Olweus conduct his research 
on bullying? 

United States Canada Norway Spain Australia 

b. According to the research by Dr. Olweus and his colleagues, who is more likely to be the 
victim of a bully? 

girls boys 

3. Imagine that your friend has a child who you think may be a bully. Write a short note to your 
friend that summarizes what you learned from the article about the causes of bullying and 
how to recognize when a child is being a bully. 



4. Based on the information in the article and your own experience, what do you think are the 
three most important things that a parent can do to prevent bullying? Explain why you think 
these three things are important. 



104 



EFF ASSESSMENT CONSORTIUM 





Improving Performance, Reporting Results 



Section 2: The Accountability Assessment Prototype for Read With Understanding 



Dealing With Bullying 



Answer Key for Answer Sheet 1 



NAME DATE 



Directions 

After you have finished reading the excerpt from the article “Bullying: 
Information for Parents and Teachers,” define the terms in question 1 
and write answers to questions 2, 3 and 4. You may look back at the 
text to find information to answer the questions, but do not copy from 
the text or from the dictionary in writing your response. Use your own 
words. You also may use knowledge that you already have about this 
subject in your responses. In some cases, there is more than one right 
answer. 



1 . Write a short definition or explanation of the following terms. Write answers in your own 
words. Do not just copy from the text or the dictionary. (Although exact wording of 
responses will vary, a proficient answer should show a level of understanding as 
indicated below.) 

a. aggressive behavior: characterized by or acting in a hostile manner; 

physically/verbally abusive 

b. retaliation: to punish or get back at someone 

c. perpetrator: a person who victimizes another person 

d. victimize: to hurt or harass someone 

e. bystander: someone who is present at or watches an event but doesn’t take part in it 

f. timid: not aggressive or bold; shy; insecure 

g. “indirect” bullying: subtle or not straightforward harassment of another person 
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2. Circle the correct answer(s) to the following questions: 

a. In which of the following country or countries did Dr. Olweus conduct his research 
on bullying? 

United States Canada ( ^orwa ^ Spain Australia 

b. According to the research by Dr. Olweus and his colleagues, who is more likely to be the 
victim of a bully? 




3. Imagine that your friend has a child who you think may be a bully. Write a short note to your 
friend that summarizes what you learned from the article about the causes of bullying and 
how to recognize when a child is being a bully. 

Answers will vary; a proficient answer will be in a note form that directly addresses the friend 
and should include information drawn from the text (such as the causes of bullying: family, 
individual and school factors) and relevant prior knowledge appropriate to recognizing 
bullying behavior in a child. 



4. Based on the information in the article and your own experience, what do you think are the 
three most important things that a parent can do to prevent bullying? Explain why you think 
these three things are important. 

Answers will vary; a proficient answer will be appropriate to the context of advice to a parent 
to prevent bullying, should include three points, and draws on information from the text and 
from relevant prior knowledge of child development of behavior. 
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SECTION 3 

Guide To Using the EFF Assessment 
Prototype for Accountability Purposes 



Getting Ready To Use the EFF Assessments 
for Read With Understanding 

Before adult education programs begin to use the accountability 
assessments in the EFF Assessment Prototype, they must take a 
number of steps to make sure the conditions are in place for fair, 
valid and reliable results: 

• Programs must have adopted curriculum and instruction aligned 
with the EFF Standard Read With Understanding. 



This section highlights the necessary 
conditions, planning and preparation 
that states, programs and teachers 
need to have in place as they get 
ready to use EFF assessments for 
accountability purposes. 



• Teachers should have received training in — and should be using 
— standards-based instruction. 

• Adult learners should have a goal to improve their reading. 

Under these conditions, program administrators then can prepare to use 
the EFF Read With Understanding assessments for measuring proficiency 
by taking the following steps: 

• Training program staff in procedures for using the EFF Read With 
Understanding performance levels for determining reading proficiency 
and placing adult learners in basic education or ESF reading classes 

• Training teachers in the use of instructional assessment aligned with the 
Read With Understanding Standard and Continuum of Performance 

to guide teaching and learning, monitor adult learners’ progress, and 
determine their readiness for postinstructional assessment using the 
accountability assessments 

State or program professional development staff may provide this training, 
with technical assistance from and/or participation of staff from the EFF 
Center for Training and Technical Assistance. 



AT THE STATE LEVEL 
Training and Preparation 

To establish a general foundation for using the EFF Read With Understanding 
assessments successfully, we recommend that state-level professional 
development providers train all program administrators and key teachers in 
adult basic education and ESF programs in standards-based instruction. This 
training should prepare these professionals to teach and assess reading for 
multiple purposes and with different measures suited to those purposes. 



In this section you will 

learn about: 

■ How teachers, adult 
education programs and 
state agencies can plan 
and prepare to use EFF 
assessments for Read 
With Understanding for 
accountability purposes 

■ How a typical classroom, 
program and state 
agency might use the 
EFF accountability 
assessments to 
measure educational 
gains for the U.S. 
Department of 
Education’s National 
Reporting System 

■ How to select, 
administer, score and 
report results of the 
EFF accountability 
assessments 
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Training in standard-based 
instruction should prepare... 
adult educators to teach and 
assess reading for multiple 
purposes... 



We also recommend specialized training for two practitioners in each 
program on how to administer and score accountabihty assessment tasks 
for Read With Understanding using procedures that are sufficiently 
standardized to ensure validity and reliability. 

Aligning Instruction With Standards 

During initial training, facilitators will need to provide teachers and 
program administrators with opportunities to deepen their understanding 
of the relationship between standards-based instruction and assessment. 



In a standards-based system, assessments actually are an integral part of teaching and 
learning, not a stand-alone activity that has httle to do with teaching and learning. 
Instruction, then, should prepare adults for standards-based assessments by 
incorporating such assessments into everyday knowledge, skill and strategy building. 
Both instruction and assessments — for placement, monitoring and accountabihty 
purposes ahke — should be based on the standards, which capture the knowledge, 
skills and strategies that are important for adults to learn. 

In planning and carrying out professional development activities about standards- 
based instruction and assessment, facilitators may make good use of EFF tools to 
accomplish this. For instance, the Read With Understanding Continuum of 
Performance can be used as a guide to evaluate evidence from intake interviews, 
informal performance observations and standardized test scores to make initial 
placements of adult learners. The onhne EFF Teaching and Teaming Toolkit provides 
ideas and resources to discover adults’ real learning goals and to conduct informal — 
but informative — assessments of reading abilities during intake interviews. And the 
EFF Teaching/Fearning Cycle points out opportunities for teachers to use EFF 
assessments for diagnostic purposes during teaching and learning. (Examples of 
instructional assessments appear on page 140.) 

In addition to this initial training, we strongly advise states to offer ongoing 
professional development opportunities, such as technical assistance to programs, 
mentoring, an onhne course for new teachers in teaching and learning with the Read 
With Understanding Standard, and meetings or institutes that focus on developing and 
maintaining reliabihty and vahdity in administering and scoring assessment tasks. 
Some guidelines and resources for such training activities are included later in this 
section. 



States may ask individual programs to submit plans for training their teachers in 

standards-based instruction. These plans can include training in strategies for: 

• Teaching to the Read With Understanding Standard 

• Administering, scoring and interpreting the results of instructional assessments 
(low-stakes assessments used during instruction to monitor learning progress and 
identify further instructional needs) 

• Administering, scoring and interpreting the results of Read With Understanding 
accountabihty assessments 
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Program plans also may describe how staff members will take advantage of 
professional development opportunities available through the state over the course 
of a year. 

We recommend requiring teachers who have been trained in administering, scoring 
and interpreting results of assessments to check their scoring periodically against other 
teachers’ scores. 

Finally, we recommend that each program designate one person (preferably the 
program administrator) to be certified as a rater of accountability assessments for Read 
With Understanding. This person will provide independent scoring (in addition to the 
scoring done by the instructor) on all accountability assessments. The certified rater 
also can assist instructors in interpreting the results of accountability assessments to 
ensure that these results are used to improve instruction. The certified rater will be 
one of the two practitioners in each program who participates in the EFF-sponsored 
training on how to administer and score accountability assessment tasks for Read With 
Understanding. Once this person is identified by the program and approved by the EFF 
staff after training, he or she will have access to ongoing technical support and 
accountabihty assessment resources from the EFF Center for Training and Technical 
Assistance. 



AT THE PROGRAM LEVEL 



Using the Continuum of Performance To Assess 
and Place Adult Learners 

We recommend that staff and teachers in adult basic education 
programs use the Read With Understanding Continuum of Performance 
as the principal tool for deciding where to place adults who are just 
entering their programs. 



Program staff should [evaluate] 
all available evidence of...abiiity 
to perform on the Read With 
Understanding Standard.. .using 



Program staff should use all available evidence of the adults’ ability Read With Understanding 

to perform on the Read With Understanding Standard — including 

Tu r T ^ T performance-level descriptions. 

interviews, tests and mtormal observations, tor example — to make 

placement decisions. We recommend interviewing adults when they 

enter the program to estabhsh their learning goals, as well as formal and informal 

assessments of their reading ability, to help to determine the level and form of 

instruction best suited to their goals and skills. 



Program staff then should evaluate the evidence using the Read With Understanding 
performance-level descriptions. Adult learners should be placed into the EFF 
performance level that is just above the performance that best matches their ability 
to read with understanding. In other words, if they appear to be able to perform 
proficiently in Performance Eevel 2, they will begin instruction on the knowledge, 
skills and strategies in Performance Eevel 3. 
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This initial placement also can be used to determine the adult learners’ corresponding 
National Reporting System Educational Functioning Level (see Table 1 on page 24, 
correspondences between EFF and the National Reporting System). Also, for some 
continuing students, posttest scores on the Read With Understanding accountability 
assessments may be used as pretest results for the next cycle of instruction. 

A Note about Pretesting for the National Reporting System 

The National Reporting System requires reporting of educational gains, defined as 
movement from one Educational Functioning Level to a higher level. This “movement” 
is determined by comparing pre and posttest scores on state-approved tests or 
benchmarks of improvement that are aligned with the qualitative descriptions of 
skills in the Educational Functioning Levels. 

For this reason, we suggest that states and programs using our Read With 
Understanding Assessment Prototype adopt an initial implementation strategy that 
includes either: 

• A standardized reading test as a pre and posttest 
or 

• State-approved, performance-level benchmarks and standardized assessments aligned 
with these benchmarks (where the benchmarks are aligned with the EFF Read With 
Understanding Continuum of Performance) 

During this initial implementation phase, we recommend using the accountability 
assessments in the Read With Understanding Assessment Prototype only for posttesting 
— and not for pretesting. There are not yet enough Read With Understanding 
accountability assessments in the item pool to support using comparable sets of items 
for both pre and posttesting. Eventually, as we add more items to the pool of secure, 
onhne accountabihty assessments, both pre and posttest use will be feasible. 



AT THE CLASSROOM LEVEL 

Instructional Assessment To Guide Teaching and Monitor Progress 

We designed the Read With Understanding accountability assessments to be used in 
programs that have aligned the content of curriculum and instruction with the EFF 
Read With Understanding Standard. With such alignment, teachers can use informal 
and formal assessments to monitor adults’ progress and improve learning and 
instruction. Teachers can use EFF tools and resources, including 
components of the accountability assessment prototype, to plan 
instructional activities that match learners’ particular goals and needs 
and to create assessments that are embedded in those learning activities. 
We will discuss guidelines and suggestions for developing some kinds of 
instructional assessments in Section 5. 

If programs initially place the adult learners in Read With Understanding Performance 
Level 2, for example, teachers can use instructional assessments designed to elicit 
evidence of performance at that level to guide the early stages of instruction. 
Instructional assessments provide opportunities for teachers to document and collect 



Instructional assessments 
provide opportunities for teachers 
to document and coilect evidence 
of an adult’s reading proficiency. 



110 



EFF ASSESSMENT CONSORTIUM 



Improving Performance, Reporting Results 



Section 3: Guide To Using the EFF Assessment Prototype for Accountability Purposes 



evidence of an adult’s reading proficiency and, at the same time, provide 
opportunities for adults to get used to their teacher assessing their reading skills 
and documenting evidence of their reading performance. 

Portfolio assessment is one way to collect and evaluate evidence of adults’ increasing 
proficiency. By developing guidelines for selecting materials to include in the portfolio, 
as well as the rubrics to evaluate these materials, teachers and adult learners can review 
learning progress together. These selection criteria and rubrics may be developed 
locally — but they should be closely aligned with the EFF Read With Understanding 
Standard and performance levels. 



Portfolios can serve multiple purposes, depending upon the guidance teachers have 
provided up front on what kind of evidence they should include. For example, adults 
might have their own portfolios in which they keep notes on reading activities and 
assessments. Teachers then can use the rubrics to evaluate materials in the portfolios to 
show adult learners — and to allow adults themselves to discover — the progress they 
are making. Together, they can discuss the reading skills and abilities they need to 
practice and improve upon. Evidence of reading progress in the portfolio may include: 

• Tape recordings of oral reading 



• Fists of materials read 

• Self-assessments of reading ability 

• Notes from teacher observations on the use of reading strategies 



Portfolio assessment is one way 
to collect and evaluate evidence 
of adult’s increasing proficiency. 



Rubrics to help adults understand how the teacher will rate their 
performance and how they can rate their own reading performance may also be 
included in the portfolio. 



As evidence accumulates in the portfolio, teachers may evaluate it to determine when 
adult learners have mastered reading proficiency at a particular performance level and 
are ready to take the Read With Understanding accountability assessments for that level. 



Guidelines for Selecting, Administering, Scoring and Reporting 

Using EFF Assessments To Report Reading Achievement 

EFF assessments provide a comprehensive measure of reading achievement to report as 
educational gains for the National Reporting System. 

As we said in Section 2, the EFF Assessment Prototype is a standardized, alternative 
assessment system designed for use in adult education. The prototype includes a 
collection of tools and guidance to help adult education programs and states assess 
educational gains in reading to meet the requirements of the U.S. Department of 
Education’s National Reporting System. These tools and guidelines also can be adapted 
and used to develop assessments on other EFF Standards. 
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The tools in the EFF Assessment Prototype include performance -level descriptions of 
key features of proficiency, assessment task specifications to guide the development of 
performance-based assessments for each performance level, a scoring rubric and model 
assessment task for each level, and an initial set of 30 performance-based assessments 
that are designed to meet National Reporting System requirements for standardized 
alternative assessment and reporting of educational gains in reading. The assessment 
prototype is a demonstration of the EFF approach to assessment for standards-based 
accountability and educational improvement. As such, the prototype can serve as a 
model that test developers can use to expand the range of assessment tools available 
for accountability and standards-based improvement. 

In the meantime, we recommend that programs use our prototype standardized 
assessments alongside standardized tests, such as the tests included in the guidelines 
for the National Reporting System — the Test of Adult Basic Education (TABE), 
Comprehensive Adult Student Assessment System (CASAS), Basic English Skills Test 
(BEST), Adult Basic and Eiteracy Education (ABEE), and Adult Measure of Essential 
Skills (AMES) — or any other standardized test approved for use in your state. 

Supplementing these standardized or state-approved tests with Read With 
Understanding assessments will produce information that can inform instructional 
improvement. While a reading score on a standardized test alone can provide a highly 
reliable measure of some aspects of reading progress, the implications of such a 
score for instructional change and improvement may be difficult to decipher. EFF 
assessments, on the other hand, are ahgned with instructional goals and content. 

EFF assessments provide information on student learning that is directly related to 
standards-based instructional objectives — and is therefore more useful to teachers 
to guide instructional improvement. Together, the two types of assessment provide a 
more comprehensive measure of reading achievement to report as educational gains 
for the National Reporting System. 

Using the EFF Read With Understanding assessments as part of a standards-based 
educational improvement strategy has significant benefits: 

• Assessments are aligned with standards. 

• Assessment results provide information about what adults know and can do that is 
useful to policymakers, program administrators and teachers. 

• Teachers can see more clearly the connections between teaching and assessing — 
and they can focus more closely on how and what to teach to maximize measurable 
learning gains. 



Selecting and Administering Accountability Assessments 

Adult learners are ready to be assessed for accountability purposes when their teachers 
determine they have mastered the key knowledge, skills and strategies for a particular 
performance level on the Read With Understanding Standard. At that time, teachers — 
with assistance from a certified rater, if needed — will select three accountability 
assessments to administer. 
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Assessment task descriptions, which are highlighted in Section 2, are available on 
the EFF Web site (http://eff.cls.utk.edu). They summarize each task’s performance 
goal, key characteristics and evidence that students will produce, as well as whether the 
task is appropriate for group or individual administration. 

The certified rater will have access to our password-protected online task 
collection. Teachers can request the full administration packets for their 
selected tasks from the certified rater — the only person in their program 
authorized to print and distribute these packets to teachers. 

To make the best use of the assessment tasks, we recommend that teachers 
follow these guidehnes: 

• Select tasks that most closely reflect the content and real-life applications 
that are relevant to the expressed goals and needs of their adult learners. 

• Make sure that adult learners are adequately prepared for the assessments 
by engaging in learning activities that build the knowledge, skills and strategies 
measured in the assessment tasks. 

• Consider the instructional context to decide whether to select assessment tasks 
that can be administered to a small group or to individuals — or some combination 
of the two. Plan classroom activities to accommodate small group or individual 
administration of the selected tasks. 

• Make sure that all the materials needed for assessment are available — the full 
administration packets, a working tape recorder and any materials listed in the 
materials section of the task description. 

• Administer all three tasks within a short period of time (no longer than one 
to two weeks). 

• Rigorously follow the step-by-step procedures included in the task administration 
packets. 



Adult learners are ready to 
be assessed for accountability 
purposes when their teachers 
determine they have mastered 
the key knowiedge, skilis and 
strategies for a particular 
performance level on the Read 
With Understanding Standard. 



Scoring Accountability Assessments 

The teacher who administers the assessment tasks will score the evidence 
of performance using the task-specific scoring rubrics provided in the task 
materials. A certified rater, working independently, should score the tasks 
again. The certified rater does not need to be present for the actual 
administration of the assessment; however, all evidence used to determine 
the level of performance on a task — such as work products, teacher 
notes, and audiotape and/or videotape — must be recorded in a form that 
permits full, independent and adequate evaluation. Thus it is important that the 
teacher make no notations of any kind on the evidence of performance. The certified 
rater must have access to all evidence and must use a scoring rubric that does not 
include any notations by the teacher. 

Note that the key goal of this scoring process is to determine whether the evidence 
justifies a rating at the “proficient” level. This should be the focus of thinking about the 



Note that the key goal of this 
scoring process is to determine 
whether the evidence justifies a 
rating at the “proficient” level. 
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evidence and using the rubric to evaluate the evidence. We suggest using this process 
for scoring: 

• Consider the evidence of performance on the assessment task as a whole and decide 
which level on the rubric (beginning, proficient or advanced) seems to be the best 
match. 

• Look at each of the performance descriptions (scoring criteria) on the rubric and 
select and mark the descriptions (at any level) that seem to best reflect the evidence. 

• Check whether these selected descriptions coincide, for the most part, with the initial 
overall performance rating (beginning, proficient or advanced). 

• Reconsider the initial overall rating in hght of the marked descriptions. Change the 
rating, if necessary. 

• Indicate a final score and write a short explanation of how the decision was made 
and how the evidence was used to justify the decision. Focus on selected descriptions 
from the scoring rubric and their connection to the evidence. 

When adult learners score proficient or higher on three accountability assessments, 
teachers and programs can report that they have moved to the next performance level. 

If the learners do not score proficient or higher on all three assessments, teachers and 
programs will need to report that they have not moved into the next performance 
level. In this case, the report will include case notes about the kinds of additional 
instruction and support that these learners will receive to prepare for retesting to 
demonstrate their mastery of their current performance level. After this short course 
of additional instruction and support, teachers will administer two new assessments to 
these learners at the same performance level at which they were assessed previously. If 
learners successfully complete these two new tasks with scores of proficient or higher, 
teachers and programs can then amend their reports to indicate that the learners have 
moved to the next performance level. 



Reporting Results of Accountability Assessments 

After the scoring process is complete, the teacher will report the results of their scoring 
to the certified rater, who will record both performance ratings (beginning, proficient 
or advanced) on an assessment reporting form and give the form to the 
program data entry clerk. 

The data entry clerk will enter the scores into a database, such as an online 
data collection and reporting system. A score is the performance level into 
which the student has advanced, such as “3” for Read With Understanding 
Performance Level 3, based on EFF assessments and state-approved 
standardized tests. To report results, the data entry clerk enters the National Reporting 
System Educational Eunctioning Eevel that corresponds to the EEE performance level. 
Here are two examples of how this works: 



Teachers and program 
administrators have direct and 
immediate access to results, 
which they can use to improve 
their programs. 
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Miranda 

Miranda takes three EFF assessments to attempt to demonstrate her 
achievement of Read With Understanding Performance Fevel 2. Her teacher 
and certified rater agreed completely on the ratings for her three 
performances: proficient, proficient and advanced. Her overall score is 
entered as a 3, indicating that she has exited Performance Fevel 2 and has 
moved to Performance Fevel 3, which corresponds to the Fow Intermediate 
Basic Education Educational Functioning Fevel of the 
National Reporting System. 

Joe 

Joe takes the same three assessments. Joe’s teacher initially rated his 
performance on all three tasks as proficient. The certified rater, however, 
independently evaluated Joe’s performance and rated one performance at 
the beginning level. After the certified rater and Joe’s teacher discussed the 
performances together and took a second look at the performance evidence, 
both agreed that one of Joe’s performances clearly merited a beginning 
rather than a proficient rating. Thus, in the end, Joe’s three performances 
are rated proficient, beginning and proficient. Because he must score 
proficient or higher on all three assessments to exit Performance Fevel 2, 
his score is entered as a 2. Case notes document that Joe will receive a brief 
period of additional instruction to build necessary skills to master 
Performance Fevel 2. After this additional instruction (which should take 
no longer than one month) Joe will take two new Performance Fevel 2 
assessments. With proficient ratings on both of these new assessments, Joe 
can be considered to have achieved Performance Fevel 2 and moved to 
Performance Fevel 3. 

Teachers or other program staff members will need to place the scoring rubrics and 
both ratings, along with all documentation and evidence of performance, in the adult 
learners’ files. Program administrators will need to make sure these files are secure and 
used appropriately. In addition, they will need to make sure the files are maintained in 
a form that permits state agencies to review them periodically — either in person or 
via samples sent through the mail. 

Adult education programs will report performance data and educational gains for all 
adult learners to the state in their year-end reports. However, teachers and program 
administrators have direct and immediate access to results, which they can use to 
improve their programs. They can identify effective (and ineffective) instructors and 
instructional practices, including teaching and learning of specific reading knowledge, 
skills and strategies. 

States will report the results from adult education programs annually to the U.S. 
Department of Education. Specifically, the National Reporting System requires 
states to report the number of adults who have demonstrated mastery of an EFF 
performance level by earning ratings of proficient or above on at least three 
accountabihty assessment tasks. 
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Putting It All Together: 

How States, Programs and Classrooms 
Can Use EFF Standards and Assessments 

So far in this guide, we’ve given you a lot of information about the EFF Standard Read 
With Understanding. We’ve explained the EFF Continuum of Performance — featuring 
the six levels of performance — and showed you how it corresponds to the National 
Reporting System. We’ve presented our Assessment Prototype and given you examples 
of the assessments we’ve developed to measure adults’ performance. We’ve taken you 
through selecting, administering, scoring and reporting results of these assessments. 

In Section 5, we’ll provide guidelines for you to develop assessments on your own. 

Now what? Flow can you put all of these pieces together into a coherent plan to use 
EFF standards and assessments to report and improve literacy results for adults in 
basic education programs? 

One State’s Story 

To help you envision how you can move forward, we’ve created three scenarios that 
show you the roles and responsibilities of practitioners at the state, program and 
classroom levels in a typical state that is using the EFF Standards now and is planning 
to use the Read With Understanding assessments. 

In the scenarios that follow, assume that State X has adopted the Equipped for 
the Future Content Standards. Key practitioners have made plans and decisions, 
estabhshed the necessary conditions, participated in EFF training opportunities, 
and taken advantage of EFF technical assistance to support programs and teachers 
effectively. Every state is different, of course, but the practices described for State X 
should be broadly applicable. 

About State X 

In 2002, State X’s adult education agency adopted the EFF Standards as the 
instructional framework for its 100 state and federally funded programs in adult 
education and English language learning. For the 2004-05 fiscal years. State X has 
decided to use the Read With Understanding assessments as a supplement to the TABE. 
State X’s long-range plan calls for using other EFF accountability tools — for Math, 
Speaking, Fistening and Writing — as they become available. Eventually, State X plans 
to use the EFF Work Readiness Credential, currently in development and slated for 
completion in 2005. 

While State X could continue to meet National Reporting Systems requirements using 
only TABE, the adult education agency — backed by its local program advisory 
committee — decided to add the EFF Read With Understanding assessments as part of 
a statewide implementation of standards-based instruction and assessment. State X 
believes that using the EFF assessments will produce these benefits: 

• Enabling program administrators and teachers to align standards, curriculum, 
instruction, assessment and reporting of performance results. 

• Providing useful information to the governor, state legislators, and business and 
industry leaders about what adults know and can do. 
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• Determining if adult learners are indeed making progress in all aspects of the 
Read With Understanding Standard. The EFF Continuum of Performance and 
performance -level descriptions make clear the knowledge and skills teachers 
should focus on for standards-based instruction and assessments. 

• Improving teaching and learning. State X knows that research shows that teachers 
who have access to good assessments and can use them well to monitor learning 
progress can be more effective in helping adult learners achieve at higher levels than 
teachers who do not make good use of assessment to guide instruction. 

Thus, State X feels confident that its investment in training teachers in standards-based 
instruction and assessment and in providing paid time for program staff to administer 
and score assessments will pay off in terms of improved teaching and learning. 

Further, teachers can select or develop their own assessments, aligned to the Read 
With Understanding Standard, to use in instructional situations to monitor progress in 
learning. With frequent assessments to diagnose strengths and weaknesses, teachers can 
keep track of and report incremental progress even when adults don’t stay in classes 
long enough to take accountability assessments. 



AT THE STATE LEVEL 

State X’s Adult Education Agency Prepares 

To Use EFF Assessments 

In Fiscal Year 2004, State X’s adult education agency puts into action a plan that will 
allow it to report educational gains to the National Reporting System using the Read 
With Understanding assessments in Fiscal Year 2005. The state adult education staff 
takes these steps: 

• Trains all program administrators and key reading teachers in adult basic 
education and ESL to teach and assess reading using the Read With Understanding 
Standard. 

Through a contract with the EFF Center for Training and Technical Assistance — 
the EFF national headquarters for professional development, materials and resources 
at the Center for Fiteracy Studies, University of Tennessee, Knoxville — and the 
state’s professional development provider (the state Fiteracy Resource Center), the 
state agency arranges a training schedule for each of the four regions of the state. 

The schedule calls for program administrators and key teaching staff to participate 
in three three-day workshops in their regions, starting in fall 2003 and conducted 
over an eight-month period. The team of training facihtators includes nationally 
certified EFF trainers and State X’s two state-certified EFF trainers. 

The trainers adapt a curriculum from the EFF/National Center for Family Fiteracy 
Reading Project, a training module that prepares teachers to integrate elements of 
evidence-based reading instruction into teaching with the Read With Understanding 
Standard. The curriculum includes practice in developing and using assessment tasks 
during instruction. In between each three-day training session, teachers try out and 
document their use of instructional and assessment tools and strategies. Program 
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administrators also gain hands-on experience in administering, scoring and 
interpreting results from assessments aligned with the Read With Understanding 
Standard so that they can provide local training and technical assistance to teachers. 

By June 2004, the program administrator and one or more teachers from every adult 
education and ESL program in the state have participated in three workshops. 

• Trains all program administrators and one teacher from every program to 
administer and score Read With Understanding assessment tasks 

The program administrator and one teacher from each program participate in the 
first two training sessions described above. For these selected participants, the third 
session focuses on scoring performance tasks, developing rehabihty between raters 
and training other teachers to score. 

• Updates the state’s online data collection and reporting system, develops 
guidelines for entering achievement data, and provides reporting forms for 
teachers who do not have access to the onhne reporting system 

State X’s online reporting system currently accepts numerical scores for selected 
assessments. To prepare the system for EFF assessments, the database programmer 
adds a field for the EFF Read With Understanding score, where the performance level 
into which the student has advanced will be recorded. The system also has a field for 
case notes, where narrative information about performance may be recorded. 

• Requires programs to report learning gains on the Read With Understanding 
Standard as part of the funding approval process 

State X’s Request for Proposals for Fiscal Year 2005 contains this language: 

“At least two staff (program administrator and one teacher, or two teachers) must 
participate in a regional training workshop on standards-based instruction and use 
of Read With Understanding assessments. 

“The program must submit a plan for training teachers in standards-based 
instruction. The plan must include training in instructional strategies for teaching to 
the Read With Understanding Standard and training in the use and interpretation of 
assessments aligned with the standard. The plan must include provisions for teachers 
to learn to administer, score and interpret the results of Read With Understanding 
instructional assessments (low-stakes assessments used during instruction to monitor 
learning progress and identify instructional needs). The plan also must include 
provisions for teachers to learn to administer and interpret the results of Read With 
Understanding accountability assessments. 

“Each program shall designate one person (preferably the program administrator) 
to be certified as a rater of Read With Understanding accountability assessments. 

This person will provide independent scoring (in addition to the scoring done 
by the instructor) on all Read With Understanding accountability assessments. The 
certified rater also will assist instructors in interpreting the results of accountabihty 
assessments to ensure that these results are used to inform instructional 
improvement. 
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“When an instructor determines that an adult learner is ready to be assessed at 
an EFF level, the instructor (with assistance from the certified rater, if needed) will 
select three Read With Understanding assessments to administer. The instructor will 
administer all assessments within a short period of time (no longer than one to two 
weeks). The instructor and the certified rater will score the evidence of performance 
on the assessments independently. 

“Both ratings, plus all documentation and evidence of performance on the 
assessments, will be preserved in a file for each adult learner in the program. These 
files will be maintained in a form that will permit periodic review by the state’s Adult 
Education Agency. The files will contain evidence of performance suitable for maihng 
to the state office of adult education; the Adult Education Agency occasionally may 
request that a sample of files be sent to the state office.” 

During Fiscal Year 2005, the State X Adult Education Agency plans to: 

• Provide technical assistance and support to programs and teachers. 

In State X, ongoing professional development and technical assistance activities are 
carried out through a contract with the state’s Eiteracy Resource Center. As part of 
this contract. State X supports two state-certified EFF facilitators — practitioners 
who have participated successfully in the EFF facilitator certification process and 
thus can conduct EFF training at the state level. 

The center’s scope of work for the year includes a schedule of technical assistance 
visits to programs, an onhne course for new teachers in teaching and learning with 
the Read With Understanding Standard, and two meetings or institutes that focus 
on developing and maintaining reliability and validity in administering and scoring 
assessment tasks. 

• Use the EFF assessments for Read With Understanding to report to the National 
Reporting System. 

The state Adult Education Agency will compile results of educational gain reports 
from adult education programs to prepare the annual report for the National 
Reporting System. The state will report the number of adult learners who have 
demonstrated mastery of an EFF performance level by earning ratings of proficient 
or above on at least three EFF Read With Understanding accountabihty assessment 
tasks. 

• Encourage programs to use data to guide program improvement. 

Adult education programs will have direct and immediate access to adult 
learner results on Read With Understanding accountabihty assessments. Program 
administrators and teachers can use these results to identify effective (and ineffective) 
instructors and instructional practices. This information can be used to guide 
improvements in instructional practices. 
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AT THE PROGRAM LEVEL 

Literacy Center in State X Incorporates Read With Understanding 

The Green River Adult Basic Education and Literacy Center, located in a midsize city 
that has a growing immigrant population and serves about 3,000 adults a year, receives 
funding from State X for its four key programs: 

• English as a Second Language 

• Adult Basic Education, including GED preparation and classes at the county jail 

• Even Start Eamily Literacy Partnerships, which help parents and children acquire 
literacy skills 

• Welfare-to-Work, which prepares welfare recipients for employment 

The Green River staff includes a director, administrative assistant, data entry clerk, 
intake and assessment counselor, three full-time teachers for the center’s four-week 
orientation class for all new adult learners, 35 part-time teachers, and 15 volunteer 
tutors in beginning literacy. Green River has a managed-enrollment policy, admitting 
new students to classes once a month in the central, primary instructional facility and 
twice a month in the satellite centers that primarily serve English language learners. 

Several years ago. Green River teachers participated in a year-long, state-sponsored LEE 
pilot, one of 25 programs in the state that explored how to use the LEE framework and 
Teaching/Learning cycle to enhance instruction. Since then. Green River has provided 
new teachers with EEL training and revised the tutor training manual to reflect EEL 
One teacher is part of State X’s EEL professional development team. So in the Green 
River programs, adult learner placement, goal setting and instructional planning all are 
informed by the LEE Gontent Eramework — the four purposes, one or more of the 
three role maps, the common activities, and the Standards. Teachers plan instruction 
using the EEL Teaching/Learning Gycle. (More information about all 16 EEE Standards, 
other components of the EEE Gontent Eramework and the EEE Teaching/Learning 
Gycle can be found on the EEE Web site at http://eff.cls.utk.edu and in many 
publications listed on 162.) 

The Green River Adult Basic Education and Literacy Genter also uses the EEE 
framework for: 

• Placing adults in appropriate programs 

When an adult learner first contacts the Green River center, the intake and 
assessment counselor schedules an intake interview. This interview follows a protocol 
that includes questions about learning goals and an informal assessment of reading, 
writing, speaking, listening and numeracy skills. 

Next, the information and observations gathered in the interview are verified using a 
relatively simple, formal test (for example, the TABE locator). Then, the Green River 
staff compares the interview and test results in reading to the EEE performance-level 
descriptions to make an initial placement decision, which establishes a baseline EEE 
Performance Level and National Reporting System Educational Eunctioning Level. 
The placement reflects the level and form of instruction best suited to the adult’s 
goals and skills. 
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• Guiding instruction 

Because the Green River center has aligned its curriculum and instruction with the 
EFF framework, informal and formal assessments that measure performance on the 
Read With Understanding Standard are appropriate for monitoring progress and 
informing learning and instruction. 

Green River staff use the Read With Understanding Gontinuum of Performance along 
with the EFF Teaching/Fearning Gycle to create in-class learning activities. Teachers 
also use EFF guidelines for developing instructional assessments, which are discussed 
in Section 5. 

• Scoring and reporting 

At the Green River center, as in all programs in State X, teachers administer and 
score the Read With Understanding accountahilitY assessments. A certified rater (in 
this case, the intake and assessment counselor) scores the assessments independently. 
Teachers are trained in the procedures for administering and scoring the assessments. 
They are required periodically to check their scoring against other teachers’ scores, 
using the EFF onhne scoring tutorial and reliability check. Additionally, the Green 
River intake and assessment counselor helps teachers administer and score the 
assessments. 

Teachers report results to the counselor, who records both ratings (beginning, 
proficient or advanced) on a Read With Understanding Assessment Reporting Form 
and turns the form in to the data entry clerk. Teachers then file the scoring rubric 
and evidence of performance in the individual files of adult learners. 



AT THE CLASSROOM LEVEL 
A Teacher Uses EFF Tools To Improve Results 

Mary Ramirez is one of Green River’s part-time ESF teachers. She teaches two classes; 
each meets for 12 hours a week. In Mary’s evening class of mixed-level learners, several 
adults who are ranked at the Low Intermediate ESF Educational Functioning Level on 
the National Reporting System are preparing for accountability assessments that will let 
them know if they are ready to move to a more advanced reading level. 

For the past four weeks, they have focused on reading about various issues and 
challenges that affect families in their community, using material drawn from the 
newspaper and from personal family stories. Mary used Level 2 on the EFF Read 
With Understanding Gontinuum of Performance, the EFF Teaching/Fearning Gycle 
and Guidelines for Developing Instructional Assessments at Level 2, along with 
accompanying examples, as guides in developing her teaching plans. She developed 
learning activities and provided instruction in a way that gave her adult learners many 
opportunities to get used to Mary assessing their reading skills and documenting 
evidence of their reading performance. 

Students have their own portfolios in which they collect notes on their reading 
activities and assessments of their reading skills. Mary uses the contents of the 
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portfolios along with the descriptions at Level 2 of the EFF Continuum of Performance 
to show individual learners the progress they are making and to discuss with them the 
reading skills and abilities they need to practice and improve upon. The kinds of 
evidence of reading progress in the portfohos include: 

• Tape recordings of oral reading 

• Fists of materials read 

• Self-assessments of reading ability 

• Notes from teacher observations on the use of reading strategies 

Each portfolio also contains a scoring rubric that Mary used to rate a Read With 
Understanding instructional assessment that she conducted recently. The learners can 
use this rubric to help them understand how Mary rates their performance on the 
standard and how they can rate their own reading performance. As evidence of 
improving reading performance accumulates in the portfolios, Mary and her adult 
learners use it along with the description of Fevel 2 on the Continuum of Performance 
to determine that the students may be ready to take the Read With Understanding 
accountability assessments that will show achievement of Fevel 2. 

Several weeks before administering the accountability assessments, Mary reviews 
the summary descriptions of the EFF Fevel 2 assessment tasks (available to teachers 
on the online EFF assessment resources). The assessment task summaries describe the 
performance goal/purpose, key characteristics of each task and the types of evidence 
that adults will produce during the task, as well as whether the task is appropriate for 
group or individual administration. Since scoring proficient or higher on three 
tasks is required to demonstrate achievement of a level, Mary selects at least three 
appropriate Fevel 2 assessments — that is, tasks that most closely reflect the content 
and real-life applications that are relevant to the expressed goals and needs of her 
learners. She submits her request for the full task administration packets to the 
Green River intake and assessment counselor in the central office. The counselor has 
a different level of access to the online task collection and is the only person in the 
program authorized to print the full assessment tasks. 

Mary chooses two assessment tasks that can be administered to a small group of 
students and one that must be administered individually. She plans her classroom 
activities for a two-week period to accommodate both the group and individual task 
administration and is careful to follow rigorously the task administration guidelines. 

Since one of the challenges of family Hfe discussed by her learners was budgeting and 
meeting expenses on a limited income, Mary chooses one task that calls for learners to 
read two simplified utility bills (one for telephone services and one for electricity) to 
understand how much is owed, when payment is due and how to make the payment. 
After reading, students must write simple one- or two-word responses to short 
questions and respond orally to questions about the utility bills. Mary administers the 
written part of the task to her five learners as a group and then schedules 10 -minute 
periods with each learner to administer the oral section of the assessment. She scores 
their individual performances using the task-specific scoring rubric that accompanies 
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the assessment. Because she periodically checks her scoring with another trained scorer 
and because she has kept up to date with training in scoring assessments, Mary is 
confident that her scores of beginning for two students and proficient for the third are 
valid and reliable. 

Mary goes on to administer all three accountability assessment tasks to her 
learners, and scores the performances within the two-week period. She submits aU 
task materials, performance evidence and “clean” scoring rubrics to her intake and 
assessment counselor, who independently rates aU the performances. Once aU scoring 
is completed, Mary arranges a time to discuss the assessment results with each learner. 
She uses the rubrics for rating performance on the EFF Fevel 2 assessment tasks as 
guides for discussing with each learner what they did well and what they need to work 
on to improve. For those learners who did not score proficient or above on aU three 
tasks, Mary plans additional instruction to build necessary knowledge and skills so that 
the learners will be prepared to do well on additional accountability assessments. 
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How the System Works at the Classroom, Program and State Levels 



Classrooms 

Teachers 

• Use the EFF tools and 
guidance to develop 
instructional activities to 
prepare adults to master 
skills and advance to higher 
performance levels 

• Administer, score and report 
results of EFF accountability 
assessments 

• Use results to plan future 
instruction with and for adult 
learners 

Adult Learners 

• Use the Standard and the 
Teaching/Learning Cycle 
during instructional activities 

• Perform on instructional and 
accountability assessments 

/V 




Local Programs 



Program Administrators 

• Supervise teachers who 
administer, score and report 
results of EFF accountability 
assessments 




State Adult 
Education Agency 



• Collect and report on adult 
learners to the National 
Reporting system in four 
required categories: 



• Maintain secure access to 
the online collection of EFF 
assessment tasks and provide 
tasks to teachers 

• Report learners’ advances in 
National Reporting System 
Educational Functioning Level 

• Use EFF goal-setting and 
achievement records to report 
achievement of National Reporting 
System secondary measure 

Database Administrators 

• Update the data collection 
system to accommodate EFF 
accountability assessments 



— Learning Gains, using 
standardized tests such 
as TABE and BEST, plus 
the EFF Read With 
Understanding Standard 

— Entry into higher 
education 

— Credentials, such as GED 
and EFF Work Readiness 

— Employment 





• Manage collection of other data 
for state, including secondary 
outcome measures on citizenship 
and family 






Professional Developers 

state Trainers 

• Train selected teachers In every program to teach and assess using 
Read With Understanding 

• Train one person in every program to monitor and support 
assessment scoring 

EFF Center for Training and Technical Assistance 

• Provides online tutorial for scoring and checking reliability of scoring 

• Provides access to online collection of assessment tasks for program 
administrators 
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SECTION 4 

Guidelines and Materials for Training 



Training People to Select, Administer, Score and Report Results 

To guarantee reliable and valid results of EFF Read With Understanding 
accountability assessments, training and monitoring need to be rigorous and 
effective. Here, we suggest key topics and practices to include in such training and 
monitoring, along with some useful training materials. This information should 
be valuable to people who develop and implement professional development 
activities in a program or state agency to support effective use of the fuU EFF 
Read With Understanding assessment system. 

We expect that the training and support offered by professional development 
providers will be targeted at teachers, other certified scorers, program 
administrators and support staff, and state agency staff, all of whom will play key 
roles in successfully implementing a statewide assessment system. Our training 
suggestions are relevant to all of these professionals. However, the primary focus 
of the information here is training for valid and reliable scoring of performance. 

Training Topics for Selecting and Administering Assessments 

Teachers need training in these topics to effectively select and administer Read 
With Understanding assessment tasks: 

1. Selecting and obtaining assessment tasks 

Facilitators will need to train teachers in several procedures for selecting and 
obtaining appropriate EFF Read With Understanding assessment tasks. Teachers 
will become famihar with summary descriptions of the tasks on the EFF Web site 
(the tasks themselves will be available on a password-protected Web page). Teachers 
will practice choosing tasks appropriate to a particular Read With Understanding 
performance level — as well as to the interests and instructional preparation of 
learners. Teachers should receive instructions for requesting full task administration 
packets from their program’s certified raters, who are the only people authorized to 
access and print out those materials. 

Facilitators also will need to train teachers to select appropriate accountabihty 
assessments by relying on their informed judgments about how well their learners 
are prepared. Beyond selecting assessments for the appropriate Read With 
Understanding performance level, teachers will need to practice choosing 
assessments based on such factors as: 

• Expressed goals and needs of their learners 

• Content they have covered during instruction 



In this section you will 
find out: 

■ How to incorporate 
standardized 
procedures and staff 
training to ensure 
the validity and 
reliability of your 
EFF accountability 
assessments 

■ What activities, 
procedures and 
materials are 
appropriate in 
training program 
administrators and 
teachers to select, 
administer, score 
and report results of 
EFF accountability 
assessments 
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• Other kinds of prior knowledge or experience that their learners might draw on in 
performance of assessment tasks 

Teachers also will need training to plan instructional activities so that they can 
administer three accountability assessment tasks over a relatively brief span of 
instructional time. Teachers will need support in thinking through how to provide 
adequate time, the appropriate environment and all necessary materials for the 
assessments — and to fit all this into regular class time. 

Finally, facilitators will need to provide teachers with: 

• Guidance in administering standardized pre and posttests approved by the state 
and in using the test scores to improve instruction 

• Information about any onsite technical assistance, online resources, and institutes 
or meetings that the state professional development provider plans to offer as 
further support for implementation beyond the initial training 

2. Getting to know the available tasks and practicing to administer them 

Facilitators will need to devote a significant amount of time to training activities 
that allow teachers to familiarize themselves with the descriptions of Read With 
Understanding assessment tasks — and to practice administering tasks by carefully 
following the procedures provided with each description. 

Training Teachers and Certified Raters To Score 

Teachers and certified raters (at least two people from each program) will need to 
participate in ongoing professional development to learn to score the Read With 
Understanding assessments. This may include: 

• Initial intensive training 

• Periodic checks on rehability of scoring and retraining on scoring, as needed 

• Regular activities that develop and maintain reliability and validity in scoring 
EFF assessment tasks 

The table beginning on the next page is an annotated outhne of key activities for 
training designed to prepare teachers and certified raters to score performance on EFF 
Read With Understanding assessment tasks. This section also provides, or refers to, 
other sources for key materials needed for this training. Most importantly, it includes 
examples of anchor evidence of beginning, proficient and advanced performances on a 
model Read With Understanding assessment task for each of the six performance levels. 
Fadhtators will use these “student papers,” along with task-specific scoring rubrics, 
primarily to help participants to differentiate levels of performance on a task and to 
develop reliability in scoring between raters. 
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Table 1. Training Activities and Materials for Scoring Assessment Tasks 


Step-by-Step Training Activities and Procedures 


Materials 


1. Review the relationship of the six Read With Understanding 
performance levels to task design specifications and scoring rubrics 
at each level. 

Divide participants into six small groups, one of each performance level. 
Each group studies and discusses one performance level and its task 
design specifications and scoring rubric. Consider: How is the performance 
level reflected In the specifications and rubric? What are the key kinds of 
information that appear in both? 


The Assessment 
Prototype for Read With 
Understanding 
(Section 2, beginning on 
page 23) 


Each small group reports its findings to the full group. 




2. Introduce the Read With Understanding scoring procedures and 
guidelines. 


Scoring procedures 
(Page 113) 


Outline general assessment guidelines to the full group. Talking points: 




• Every performance will be scored twice — once by the teacher 
administering the assessment and a second time by a certified rater. 
If two raters produce different final scores, these raters will need to 
work together to understand each other’s reasoning and negotiate an 
agreement on the final score. 




• Teachers will score individual performances using a task-specific scoring 
rubric, which describes the evaluation criteria for the assessment. 




• A second certified rater will score Individual performance Independently, 
using the same scoring rubric. To make this possible, teachers must 
record all performance evidence in a form that permits full, independent 
and adequate evaluation by a reviewer who is not present at the time 
that the task is administered. Assessment administration and scoring 
procedures outlined in each assessment task description should be 
followed carefully so that the second rating of the task is in fact 
independent. 




• The two scores for each performance are compared and negotiated if 
necessary. 




Divide participants into small groups to review the scoring procedures. Ask 
participants to read scoring procedures individually and then discuss in 
small groups, noting any comments or questions and reporting to full group. 
Organize and address key questions. 
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Table 1. Training Activities and Materials for Scoring Assessment Tasks, continued 

Step-by-Step Training Activities and Procedures Materials 



3. Get familiar with the Read With Understanding scoring rubric. 

Walk through a sample scoring rubric with the full group. Talking points: 



One scoring rubric (from 
Section 2, beginning on 
page 35) 



• The scoring rubric states the criteria used to evaluate performance as 
beginning (just below what Is required for level achievement), proficient 
(required for level achievement) and advanced (clearly superior to what is 
required for level achievement) 



• The most critical scoring imperative is for the teacher or certified rater to 
link, as directly as possible, each performance criterion to some piece of 
performance evidence 



4. Make an initial scoring decision. 

Give each participant one copy of a Read With Understanding Performance 
Level 3 task description; a clean scoring rubric; and one example each of 
beginning, proficient and advanced performance evidence (randomly 
labeled A, B and C, not labeled as beginning, proficient or advanced). 



Model task and scoring 
rubric for Read With 
Understanding 
Performance Level 3 
(From Section 2, beginning 
on page 57) 



Ask participants to take a few minutes to read the task description 
and scoring rubric on their own, then briefly look over the three sets of 
performance evidence. Ask them to decide which performance is weakest 
and mark that with a 1 , which performance is strongest and mark that with 
a 3, and which falls in the middle and mark that with a 2. Tell participants 
they don’t need to make any explicit connections to the scoring rubric yet. 



Performance evidence 
randomly labeled A, B and 
C representing Beginning, 
Proficient and Advanced 
scores on Performance 
Level 3 (on EFF 
assessment Web site) 



Then, in the full group, lead a discussion of this activity. Taking one set of 
performance evidence at a time, ask “How many scored this as weakest? 
stronger? strongest?” Mark the number of responses on a simple scoring 
matrix drawn on chart paper. Then ask a few participants to explain why 
they chose the scores they did (in each category if there is disagreement). 
Ask, “Given these explanations, would anyone want to change their 
scores?” Revise the matrix as necessary. Repeat with the other two sets 
of performance evidence. 



Scoring matrix 
(Figure 3, page 131) 
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Table 1. Training Activities and Materials for Scoring Assessment Tasks, continued 

Step-by-Step Training Activities and Procedures Materials 



5. Connect scoring criteria to performance evidence. 

Hand out to the full group the model task; three sets of performance 
evidence scored as beginning, proficient and advanced; scored rubrics; 
and any other reievant scoring artifacts for Read With Understanding 
Performance Levei 4. 

Ask participants to take a few minutes on their own to read the task 
description. Then iead a fuii-group discussion about the performance 
evidence and scoring artifacts for proficient. Paying close attention to 
the rubric and the evidence, taik about why this is performance is scored 
proficient. Point to specific ianguage in the scoring rubric that connects to 
specific evidence in the performance. Then ask, “Do you agree that this is 
a proficient performance? Why or why not? Does the scoring raise any 
questions that we need to pay attention to?” 

Divide participants into smali groups. Ask each group to use the same 
process to look at the beginning and advanced performances. Ask them 
again to pay close attention to the language of the scoring rubric and how 
it connects to the performance evidence. Ask them to decide if they agree 
with the score. Address any questions or concerns that the scoring raises. 



Model task and rubric for 
Read With Understanding 
Performance Levei 4 
(from Section 2, beginning 
on page 66) 

Performance evidence 
randomiy labeied A, B and 
C representing beginning, 
proficient and advanced 
scores on Performance 
Level 4 (on EFF 
assessment Web site) 

Scoring matrix 
(Figure 3, page 131) 
Materiais 



With the full group, lead a debriefing of this activity, asking: “How did 
it go? Did you agree with the scores? Why or why not? What questions 
or concerns do we need to pay attention to?” 

Issues that ought to be addressed In this activity, whether or not 
participants raised them: 



• The potentiai for inflating scores when teachers administer assessments 
and really want their student to do well, which points to the critical role of 
solidly connecting scoring criteria to performance evidence to make sure 
scoring is fair and reliable 



• Using a holistic approach to scoring to make tough decisions when the 
performance evidence does not all neatly fall into one level and where a 
decision needs to be made based on the overall weight of the evidence 
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Table 1. Training Activities and Materials for Scoring Assessment Tasks, continued 

Step-by-Step Training Activities and Procedures Materials 



6. Practice scoring — Phase 1. 

Ask participants to work in pairs to repeat the previous exercise with a 
model task and three levels of performance evidence, plus scoring artifacts, 
for Read With Understanding Performance Level 5. Then have each pair join 
with another pair to discuss their findings and identify questions. 

Lead a debriefing discussion with the full group and field questions. 



Model task and rubric for 
Read With Understanding 
Performance Level 5 
(from Section 2, beginning 
on page 80) 

Performance evidence 
randomly labeled A, B and 
C representing beginning, 
proficient and advanced 
scores on Performance 
Level 5 (on EFF 
assessment Web site) 



7. Practice scoring — Phase 2. 

Ask both participants from each program to work together. Have each 
pair choose one of the three remaining model assessment tasks. Direct 
everyone to work independently, following the scoring procedures, to rate 
three performances on that task. When everyone is finished, ask participant 
to team up with their program partners to compare scores and justifications 
for their scores. Talking points: 

• If partners disagree on any of the scores, they negotiate with each 
other to come to an agreement on a final score, always focusing on the 
connection between the rubric criteria and the evidence. 



Scoring matrix 
(Figure 3, page 131) 



Model tasks and rubric for 
Read With Understanding 
Performance Levels 1 , 2 
and 6 (from Section 2, 
beginning on page 34) 

Performance evidence 
randomly labeled A, B and 
C representing beginning, 
proficient and advanced 
scores on Performance 
Levels 1 , 2 and 6 (on EFF 
assessment Web site) 



• It would be great — but is not mandatory — for partners to agree at 
this point. They do, however, need to be able to explain their process 
for trying to come to consensus. 



Scoring matrix 
(Figure 3, page 131) 



Next, ask each pair to join with other pairs who are working on the 
same task. (There should be three groups, one for each task.) Ask them 
to discuss their process to try to agree on scores (whether agreement 
was achieved or not), and identify questions and concerns. 

Then, lead the full group in a debriefing of this activity and field questions. 
If there is time, let participants work through another task in this way. 
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Scoring Procedures for Assessment Tasks 

• Consider the evidence of performance as a whole and make an initial decision about 
which score on the scoring rubric (beginning, proficient or advanced) best describes 
the evidence. In other words, decide which score is the best “match.” 

• Look closely at the language on the entire scoring rubric — the scoring criteria — 
and highlight the words or phrases for any criteria that seem to best reflect the 
performance evidence. 

• Check to see whether your highhghted language appears primarily in the score you 
initially selected as the best match for the evidence of overall performance. 

• Reconsider your initial decision; decide whether to keep that score or change the 
score. 

• Write a short explanation on the scoring rubric justifying your decision. Describe 
how you decided on the score, how the evidence was used to support it, and how 
the scoring criteria on the rubric is related to features of the performance evidence. 

• Make sure that the scoring rubric and all performance evidence is collected, labeled 
with the adult learner’s name or initials, filed appropriately in the individual’s file, 
and stored in a secure site. 



Figure 3. Scoring Matrix 



Weakest 


Stronger 


Strongest 


Performance A 








Performance B 








Performance C 









Training on Reporting and Interpreting Results of Assessments 

To allow for reporting of results of EFF Read With Understanding assessments, the state 
agency responsible for adult basic education will need to adapt its online reporting system, 
publish guidelines for entering data and develop paper reporting forms for people who 
cannot report online. 

At this point, providers of professional development will need to train people in programs to 
make sure reporting is accurate and complete for state and federal accountabihty purposes. 
Specifically: 

• Each program’s certified rater will require training and practice in completing the EFF Read 
With Understanding Assessment Reporting Form that includes information submitted by 
program teachers. 

• Each program’s data entry staff will need training to learn about changes to the state’s 
online reporting system, along with practice using the EFF Read With Understanding 
Assessment Reporting Form to complete the program’s report about educational gains 
for the state agency, following the state-published guidelines for entering data. 
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SECTION 5 

Guide To Using EFF Assessments 
To Improve Teaching and Learning 



The need for aligning 
instructional and assessment 
content applies not only to 
assessments used at the end 



Designing and Using Instructional Assessments 

In Section 1, we sketched out the theoretical foundation of the EFF Assessment 
Prototype: Aligning accountability assessments with curriculum and instruction 
is a basic requirement for the theory of action of standards-based educational 
improvement to work in practice. In other words, accountabihty 
assessments can provide valid, fair and useful information on 
instructional program results only when such assessments are a good 
fit with what learners have been taught and what they have had an 
opportunity to learn. 



of a course of instruction for 
accountability reporting, but 
also to instructional assessments. 



This need for aligning instructional and assessment content applies 
not only to assessments used at the end of a course of instruction for 
accountability reporting, but also to instructional assessments (sometimes 
called instructionally embedded assessments or classroom-based 
assessments) that are used during instruction to monitor progress 
and guide teaching and learning. 



In this section you will 
find out: 

■ How teachers can 
use EFF tools and 
resources to design 
and administer their 
own assessments — 
aligned with EFF 
Standards and 
accountability 
assessments — to 
improve teaching 
and learning 



All of the 16 EFF Content Standards focus on performance — that is, 
on applying knowledge and using strategies in real-hfe situations. When the 
EFF team was still field-testing and refining these standards, we recognized that 
comprehensive assessment of performance on the real-world, applied skill 
processes embodied in the Standards 
would require teachers and students to 



become more famihar with performance- 
based assessments and to begin to 
integrate such assessment into 
instructional activities. 

To that end, we published EFF Assessment 
Report: How Instructors Can Support 
Adult Fearners Through Performance- 
Based Assessment (Ananda, 2000). This 
report introduces adult educators to the 
whys and hows of performance-based 
assessments in instructional settings. 

We wanted to help practitioners more 
effectively align the content of teaching 
and learning with the EFF Standards 
and with performance assessments on 
the Standards. The report guided 



“Performance-based assessment 
is an approach to assessment 
that is congruent with 
EFF’s emphasis on real-world 
performance. ... Performance- 
based assessment includes 
cognitively demanding, hands-on 
activities. It aims to stimulate 
learners to think; react to new 
situations; review, revise and 
evaluate their work; and 
communicate in verbal and visual 
ways.” (Ananda, 2000, p. 3) 
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practitioners in developing instructional assessments that link to 
individual EFF Standards — as well as to the adult learners’ needs and 
goals. In these ways, the report promoted the practice of standards-based 
educational improvement. 

The EFF Assessment Report was essentially an introduction. Now, with 
the EFF Read With Understanding Assessment Prototype — which, as we 
have shown, includes the Continuum of Performance, performance -level 
descriptions, assessment task design specifications and model assessments 

— we offer a working model for one standard of the full range of tools 
that practitioners will need to effectively ahgn the content of their 
curriculum and instruction with assessment of performance on the 
EFF Standards for a variety of purposes. 

Beyond accountability purposes, which we have discussed at length in this 
guide, another important way for teachers to use our tools for alignment 

— and so, for standards-based educational improvement — is to develop 
and use EFF instructional assessments. (See Table 2 below.) 



“Performance-based assessment 
purposefully blurs the lines 
between teaching, learning 
and assessing. At times, an 
assessment can serve simply as 
a gauge of student progress. 

At other times, it can also serve 
as a powerful instructional tool, 
providing meaningful learning 
experiences in itself.” 

(Ananda, 2000, p. 7) 



Table 2. Comparing Instructional and Accountability Assessments 

Examples of 

Instructional Assessments Assessment Tools 



■ Incidental Assessment 

Incidental assessment captures unplanned evidence of learning and 
diagnosis of learning needs that teachers or learners notice during 
instruction. Effective teachers keep a record of this evidence to heip 
them plan future activities. 

As iearners progress, teachers coiiect this kind of evidence and review 
it in iight of the Standard and performance-levei descriptions. They think 
about how the evidence illustrates (or doesn’t) learners’ progress within or 
between performance levels. 



EFF Standard and performance- 
level descriptions 

• Teacher questioning, 
conversations with learners 

• Teacher observations, jotted notes 

• Teacher logs 

• Learner logs 

• Unplanned sample work kept in a 
learning portfolio 



■ Ongoing Assessment 

Ongoing assessment takes place during the teaching and learning process, 
as reflected in the EFF Teaching/Learning Cycle. It includes any kind of 
planned activity focused on an EFF Standard used to gather evidence of 
learner progress, to diagnose learning needs, or to reflect on learning. 
Teachers, learners themselves and sometimes peers collect the evidence. 
Teachers and learners review the evidence in light of the Standard and 
performance-level descriptions. This type of review helps learners take 
responsibility for their own learning. 



EFF Standard and performance- 

level descriptions 

• Brainstormed lists of learning 
needs 

• Learner self-evaluation worksheets 

• Oral interviews 

• Written essays or quizzes 

• Videos of student work 

• Portfolios of ongoing classroom 
work evaluated informally using 
assessment rubrics and checklists 
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Table 2. Comparing Instructional and Accountability Assessments, continued 

Examples of 

Instructional Assessments Assessment Tools 



■ Instructional Tasks 

More formal performance-based assessments are planned In advance 
and include more structured rubrics, checklists or other assessment 
tools to gather evidence of iearner progress on performance of an 
EFF Standard. In designing the assessment activities and tools, 
teachers refer to the EFF Standard, performance-level descriptions and 
assessment task specifications. Often these instructional tasks are used 
at the end of a teaching and learning cycle to see if adult learners are 
ready for more formal assessment for accountability purposes and to 
allow them to gain experience with performance-based assessment 
tasks. 



EFF Standard and performance- 

level descriptions 

• Guidelines for instructional 
assessment 

• Planned description of the steps 
of the task 

• Observation guides, checklists, 
rubrics 

• Process for sharing results of the 
assessment with learners 



■ Accountability Assessment Tasks 



EFF Standard and performance- 
level descriptions 



EFF accountability assessment tasks are similar to instructional tasks in 
that they both measure the same knowledge, skills and abilities defined 
by the EFF Standard and they both allow for the gathering of evidence 
of performance on the standard. 

However, there are some important differences as well. First, 
accountability assessment tasks are more highly standardized to 
ensure the reliability and validity of the results. The tasks themselves are 
“secure,” which means that adult learners may not see them in advance 
and only teachers who are trained to assess students for accountability 
may access them. The tasks have been field tested to ensure their 
reliability and validity. Tasks are administered and rated by teachers 
and a certified rater specifically trained to score tasks reliably. Results 
are used to measure learner progress from one level to another on the 
Standard and make decisions about moving students to another level or 
class. And because of the standardization in task design, administration 
and scoring, the results can be compared across students, programs 
and states. 



EFF assessment design 
specifications and modei 
assessments (Read With 
Understanding) 

• Guidelines for administering 
EFF assessment tasks 

• EFF task description based on 
specifications for the appropriate 
level 

• Formal observation guides, 
checklists, rubrics or other tools 
based on the Continuum of 
Performance for the Standard and 
task specifications 

• Guidelines for reporting scores 
for local, state and/or federal 
reporting 
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What Are EFF Instructional Assessments? 

EFF instructional assessments, as described in Table 2, may vary in form, but their 
primary purpose is to provide feedback on learning progress that can be used to guide 
further learning and instruction. 

EFF instructional assessments may be relatively formal assessments of performance 
that consist of activities that are mostly distinct from instructional activities (see 
example below). Or they may be informal assessments that are so closely interwoven 
with instructional activities that they are indistinguishable from instruction. These 
integrated (or embedded) instructional assessments may involve evaluation of 
performance on an EFF standard based on evidence that is observed or collected in the 
normal course of instructional activities. EFF instructional assessments can be used to: 

• Gather evidence about learners’ progress on an EFF Standard at a particular level 

• Help learners to understand how progress on the Standard is evaluated 

• Aid in planning further instruction 

EFF instructional and accountability assessments are similar in that both are developed 
using specifications that are tied to the EFF performance-level descriptions and both 
are used to produce and evaluate evidence of proficient performance on a targeted 
EFF Standard. The main difference is that administering and scoring procedures for 
low-stakes instructional assessments — unlike accountability assessments — need 
not be standardized. Therefore, teachers can develop their own EFF instructional 
assessments and tailor their content, procedures and scoring to their learners’ specific 
purposes and goals. Using instructional assessments as a regular component of the 
teaching/learning process not only builds learners’ expertise in use of the Standard, 
but also prepares them to perform well on accountability assessments. 

Using Instructional Assessments: The EFF Teaching/Learning Cycle 

The EFF Teaching/Fearning Cycle provides a framework to help instructors plan 
when and how to assess performance on the EFF Standards and how to use assessment 
information to guide teaching and learning. 

EFF staff and field researchers developed the eight-step EFF Teaching/Fearning Cycle 
in response to questions from teachers and others about what teaching, learning and 
assessment in a standards-ahgned system might look like in practice. The Teaching/ 
Teaming Cycle provides guidehnes that can be adapted by teachers in a variety of 
contexts. The basic steps of the cycle are described in Table 3. More details, examples 
from practice and a Teaching/Fearning Toolkit of resources for teachers can be found 
at http://www.cls.coe.utk.edu/efftlc/ 
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Table 3. Steps in the EFF Teaching/Learning Cycle 



step 1 Determine the adult learner’s goals and purposes and Identify the Standards 
that will help him or her achieve them. Identify the learner’s prior knowledge 
about the goals and standards. 



Step 2 Identify a shared interest, purpose or goal and determine the group’s prior 

knowledge of the topic. Identify the standard that will help the group address 
the shared goal. Make clear the connection between the class focus and 
Individual learners’ needs. 



Step 3 


Design a learning activity to address the real-life concerns of the learner(s). 


Step 4 


Develop a plan to capture evidence and report learning. 


Step 5 


Carry out the learning activity. 


Step 6 


Observe and document evidence of performance of the standard. 


Step 7 


Evaluate and reflect on how what was learned is transferable to real-life 
situations. 



Step 8 Determine next steps to help learners meet their goals. 



As illustrated in Table 3, the instructional process begins with finding out individual 
adult learners’ interests and immediate real-world goals through intake and goal- 
setting activities that take place in the classroom. (See the EFF Teaching/Fearning 
Toolkit for suggested activities). As part of this process, adult learners and teachers 
together identify the EFF Standard that will be the focus of instruction (Step 1). 

During this process, learners begin to become familiar with the Standard, how the 
Standard can help them to meet real-world goals, and what they will be expected to 
know and do to meet the Standard. This is reinforced as they identify their prior 
knowledge of and experience with activities related to their goal and to the Standard 
(Step 2). Fearners identify “what we already know/can do” and “what we still need to 
find out/be able to do.” Using the key knowledge, skills and strategies identified in the 
EFF Continuum of Performance at the appropriate performance level of the Standard, 
they identify “gaps” in prior knowledge/experience that will be the focus of instruction. 
This process allows learners to begin the learning process with an understanding of 
what they will be expected to learn and of what they will be assessed. 
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whenever possible, adult learners are also involved in developing a learning activity or 
series of activities to help them to meet their goal (Step 3). At this stage, teachers often 
use a planning guide to help them answer three key questions: 

• What do we need to know? 

(A checklist of learning goals based on the standard) 

• How can we learn it? 

(A description of learning activities) 

• How will we show we know it? 

(Assessment activities) 

For each learning activity, students also are asked to think about how they will 
demonstrate what they have learned, both to themselves and to others (Step 4). This 
process of ongoing assessment can be connected closely to the learning process. For 
example, students may write reflections on what they have learned daily in a personal 
journal. They may report on an activity they did at home with their child or use a 
checklist to evaluate their work as they are writing. They may keep summaries or 
action plans in a portfolio. AU of these assessment activities may be used as part of an 
ongoing and iterative instructional process. (See the EFF Teaching/Learning Toolkit 
for ideas for ongoing assessment.) 

As learners develop and practice their knowledge and skills, instructional assessments 
can be used to allow them to see how far they have come in mastery of the Standard at 
a particular level (Step 6). 

Toward the end of the cycle, learners are encouraged to reflect on what they 
have learned from the learning activities, their ongoing assessment work and their 
participation in any instructional assessments that were part of the activity (Step 7). 
This step allows them to evaluate what they have learned, how it can help them to 
meet their real-world goals, and how their learning might transfer to other life roles 
and situations. 

Then, teachers may consult with learners to make decisions about whether the learners 
are ready for the EFF accountability assessments that will document achievement of a 
level of performance and readiness to move into a higher level (Step 8). At this stage in 
the cycle, teachers and learners also may decide that more instruction and learning is 
needed before accountability assessments are administered — or they may decide to 
focus their work on a new Standard or Standards. 



Using the EFF Specifications for Accountability Assessments 
To Develop Instructional Assessments 

To create their own instructional assessments that are aligned to the EFF Standards 
and accountability assessments, teachers can use the same design specifications as 
those used to create accountabihty assessments, which are tied directly to the EFF 
performance-level descriptions. (Design specifications for EFF accountability 
assessments begin on page 34 in Section 2.) 
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We offer several guidelines to assist teachers in this endeavor. While these guidelines are 
written to be consistent with developing a relatively formal assessment, they are meant 
to help teachers think through two key points in designing any type of assessment — 
gathering sufficient performance evidence and then scoring it fairly and appropriately. 

• Consider the issue, problem or topic that is, or recently has been, the context for 
teaching and learning to a targeted standard at a particular level. Then think about a 
meaningful, real-hfe task related to that issue, problem or topic that learners will care 
about accomphshing, that is doable in the instructional setting and that will allow 
learners to produce evidence that they have mastered (or have not yet mastered) the 
targeted standard at the appropriate level. 

• Develop the details of the assessment task by referring to the design specifications 
for accountabihty assessments at the corresponding level. 

• Use the design specifications labeled “Characteristics of Assessment Tasks” and 
“Evidence” to: 

— Clearly identify (and share with learners) the performance goal of the task: 

Why and in what meaningful context will learners use the targeted EFF 
Standard in this task? 

— Outline the major activities or components of the task: What will learners do? 

— Decide what kind of performance evidence learners can generate during the task 
and how that evidence will be documented and collected. 

— Identify any texts, tools or other materials (including answer sheets, observation 
forms and so on that teachers will need to develop to collect performance 
evidence) that learners will use to perform the task. 

— Plan for where learners will perform the task and approximately how much time 
they will need to complete it. 

• Use the design specifications labeled “Knowledge and Strategies Assessed at 
Performance Eevel ...” to review and clarify how the instructional assessment will 
illustrate proficient use of the standard: 

— Are the content and vocabulary for the task appropriate to the level? 

— Does the task require learners to apply knowledge and use strategies that are 
appropriate to the level? 

— What else will learners need to know and/or be able to do — beyond the scope of 
assessment — to successfully complete the task? Do the learners have sufficient 
prior knowledge or have they been adequately prepared in these additional areas? 

• Use the design specifications labeled “Step-by-Step Procedures for Administration” 
to: 

— Develop a “script” for the task. This can be as formal or informal, as flexible 
or precise as the teacher believes is necessary. The key is to make sure that all 
learners get the same clear message about what is expected of them — and equal 
opportunity to perform. 

• Use the “Scoring Rubric Template” to: 

— Develop a scoring rubric that is customized so it is appropriate for scoring the 
evidence that will be produced in the instructional assessment. 



138 



EFF ASSESSMENT CONSORTIUM 



Improving Performance, Reporting Results 



Section 5: Guide To Using EFF Assessments To Improve Teaching and Learning 



Procedures for Administering Instructional Assessments 

Administering instructional assessment tasks can be less standardized 

than administering assessments for high-stakes accountability purposes 

in a number of ways. For instance, in an instructional setting: 

• It is not essential for teachers to read the script for a particular task word 
for word. The script can act simply as a guide. What is most important 
is that learners get a very clear and consistent message about what they 
are expected to know and be able to do to perform proficiently. 

• Teachers may feel free to share the scoring rubric for the task with 
learners, in whatever way makes sense, before they begin to perform 
the task. In fact, reviewing the rubric with learners is a useful way to be 
explicit about how performance of the task will be assessed and what 
“good enough” performance looks Hke. 

• Teachers may, if they think it necessary, ask learners additional questions 
or use other prompts while they administer the task beyond what is 
suggested in the task procedures to urge learners to explain something 
further or draw out more information. They also may engage learners 
in discussion about activities within the task as learners are performing 
them. This may be especially helpful if such a discussion will yield 
evidence of learners’ ability to link new information to prior experience 
or to identify and monitor their use of cognitive or metacognitive 
strategies. 

• In some cases, teachers may be able to change or adapt a particular 
activity or component of a task to better suit the learning styles or 
interests of learners. The key concern needs to be ensuring that the 
changed or adapted part of the task is still useful for collecting the 
appropriate evidence of performance. 

• Some tasks may be designed so that learners can assess the progress of their 
classmates. When used with sensitivity, this can be a good way to help learners 
develop a deeper understanding of the scoring rubrics and the rating process. 

• Teachers and learners can debrief performance of the task together as soon as 
they finish. Teachers can share scores and observations to give timely feedback to 
learners on how well they used the standard to perform the task, what knowledge 
and strategies they used well, and what they struggled with. Learners also can use 
the scoring rubric to reflect on and assess their own performance. Because the 
performance information is immediate and transparent, teachers and learners can 
use the data before them with confidence to plan for next steps in standards-based 
instruction. 



The sample instructional 
assessment task that begins on 
page 140 exemplifies a task that 
would be meaningful to many 
adults in basic education, 
especiaily in classes focused 
on workforce preparation and 
employment. The task, 
“Determining Minimum-Wage 
Rights,” gives evidence of how 
a teacher can use the EFF 
Teaching/Learning Cycle, along 
with these guidelines, to create 
an instructional assessment 
task that would be useful for 
diagnosing and monitoring 
student progress and teaching 
effectiveness. 
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Sample Instructional Assessment Task and Scoring Rubric 


EFF Read With Understanding Performance Level 3 


Determining Minimum-Wage Rights 


Characteristics of 
Assessment Task 


Task Description 




Action — Performance Goal 


Read an edited version of “Your Rights Under 
the Fair Labor Standards Act,” the poster about 
minimum wage that all employers are required 
to display in the workplace, to determine if a 
job wage is legal and, if not, what response the 
employee should make. 




Type of Text 


Environmental print (printed words found in the 
real world) featuring sections from the poster 
with several short, simple paragraphs. Scenario 
with simple sentence structures, familiar 
content. 




Content 


Everyday vocabulary; some specialized, 
employment-related vocabulary 




Environment 


Familiar environment (classroom or employment 
resource room) 




Estimated Time 
To Complete Task 


One hour 




Materials 


• Environmental print (“Your Rights Under the 
Fair Labor Standards Act,” in English, edited 
so that the section on child labor is deleted 
but the rest of the actual text remains) 

• Scenario describing an employee who has 
recently started a job. 

• Answer Sheet, Assessor Observation Form 
(on page 149) 

• Dictionary 
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Sample Instructional Assessment Task and Scoring Rubric 

EFF Read With Understanding Performance Level 3 



Determining Minimum-Wage Rights 



Task Description 



Knowledge and 
Strategies Assessed 

Adults performing 
at Performance Level 3 will 
fluently and independently 
use this knowledge and 
strategies in this task 



• Decoding and recognizing most words in 
short to medium-length continuous text by 
drawing on content knowledge and oral 
vocabulary, breaking words into parts, 
applying pronunciation rules, and adjusting 
reading pace. 

• Demonstrating familiarity with common, 
high-interest content knowledge and related 
vocabulary 

• Locating important information in text 

• Monitoring and enhancing comprehension by 
using a range of simple strategies, such as 
recalling, restating, rephrasing, explaining the 
content of the text or using simple examples 

• Actively applying prior knowledge to assist in 
understanding information in texts 



Additional knowledge and 
strategies that may affect 
performance on this task 



• General familiarity with work in the United 
States, the concepts of minimum wage and 
workers’ rights, and how hourly workers are 
compensated (hourly rates, pay schedules) 



Evidence 



Observations 



Observations of oral reading and explanation 
of reading content documented on Assessor 
Observation Form 



Work Products 



Written responses 
Sheet 



to questions on Answer 
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Sample Instructional Assessment Task and Scoring Rubric 

EFF Read With Understanding Performance Level 3 



Determining Minimum-Wage Rights 



Task Description 



Instructional Context 



This task may be appropriate for an 
adult iearner or a group of iearners who are 
interested in developing their ability to read 
with understanding to iearn about empioyment 
issues. Thus, before beginning this activity, the 
teacher and learner(s) will already have engaged 
in goai-setting activities that identify this interest. 



This section does not 
appear in a description 
of an EFF accountabiiity 
assessment task. 



The teacher engages learner(s) in caiiing to mind 
prior knowledge and experience with working 
and getting paid in the United States (inciuding 
the concepts of hourly wage and of workers’ 
right to be paid at least minimum wage in 
“conventional” workplaces and related 
information and vocabulary). This might be done 
in a structured conversation or interview, with 
a question-and-answer sheet, through guided 
responses to pictures or simple reading prompts, 
and so on. 



The teacher and learner(s) explicitly identify “what 
we already know” and “what we need to find out” 
about workers’ right to be paid at least minimum 
wage. To complete this task successfully, 
learners will need to be familiar with: 

• Concepts of hourly and minimum wage 

• General information on the role of law and the 
U.S. Department of Labor in regulating work 
wages 

• Everyday and some specialized employment- 
related vocabulary like that used on the poster 
and in the scenario 

• Uses and examples of “environmental print” 

• How to use a dictionary 

Learners also will need to be taught — and 
to have practiced — strategies for locating, 
rephrasing and explaining in their own words 
specific information found in short sections of 
text. 
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Sample Instructional Assessment Task and Scoring Rubric 

EFF Read With Understanding Performance Levei 3 



Determining Minimum-Wage Rights 





Task Description 




Step-by-Step Procedures 


The oral reading and responses are 




for Administration 


administered individually. Silent reading and 
written responses may be administered in 
small groups. 

There are two main parts of this task: 

1 . Reading and responding to a poster on 
minimum-wage rights 






2. Reading a scenario and applying the 


This scenario might 




information on the poster to the situation 


be repiaced with a 




described in the scenario 

Provide learners with an edited graphic 
representation of the poster “Your Rights 
Under the Fair Labor Standards Act,” as 
well as a short, direct scenario describing 
an employee who has recently started a job 
at below minimum wage. 

Part 1 of the Task 


roie-piaying script, 
described beiow. 


Step 1 


Explain the task requirements and 


Note that in an 




expectations; “This task will assess how well 


instructionai setting the 




you can read and understand information on a 


teacher does not need 




poster about workers’ right to be paid at least 


to stick to the script 




minimum wage, and then relate that information 


as iong as iearners get 




to a real-life story. The poster is called ‘Your 


a very dear message 




Rights Under the Fair Labor Standards Act.’ 


about what they are 




It describes a worker’s legal right to be paid 


expected to know and 




minimum wage by an employer. The story 
describes a woman who has recently started 


be abie to do. 




a job. You will be asked what you feel are the 


Note aiso that the 




important points of the story and how these 


teacher may share the 




points relate to the information on the 


scoring rubric for the 




minimum-wage poster.” 


task with iearners, 
reviewing it as another 


Step 2 


Ask the learner to spend a few minutes 


way to be expiicit about 




reading the poster: “Please read this poster 


how performance of the 




to yourself. If you need help with any of the 


task wiii be assessed and 




vocabulary, feel free to use this dictionary. If 


what a “good enough” 




you want, you may write down any points that 
are new to you or that you think are important.” 


performance iooks iike. 
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Sample Instructional Assessment Task and Scoring Rubric 

EFF Read With Understanding Performance Level 3 



Determining Minimum-Wage Rights 



Task Description 



step 3 



Give the learner an Answer Sheet to identify, 
in writing, important points made in the 
poster: “Please use this Answer Sheet to write 
down in your own words: 

• Two important points made on the poster 
about wages 

• Two important points made on the poster 
about enforcement of the law 

Please do not just copy what is on the poster, 
but use your own words.” 



Step 4 



Ask the learner to read the top section 
of the poster aloud and discuss the two 
important points identified in the section: 

“Please read the top section of the poster to 
me out loud, up to the words ‘Overtime Pay,’ 
and then read or tell me about the two 
important points you wrote down about this 
section.” Document observations on the 
Assessor Observation Form. 



Step 5 



If necessary, ask the learner to further 
explain the reading (beyond the two points 
identified) and use questions to draw out 
information, such as: 

• “According to the information on the 
poster, do minimum-wage laws apply 
to all employees?” 

• “Who can legally be paid less than minimum 
wage?” 

• “If you are a waiter and you get tips, what 
does the law say about your right to minimum 
wage?” 

Documents any evidence of understanding of 
the reading on the Assessor Observation Form. 



This step would not be 
included in the more 
standardized procedures 
of an EFF accountability 
assessment task. 



Then engage learners in discussion about the 
information on the poster and how it might 
relate to their lives: 

• “Can you think of a time when you could have 
used this information?” 

• “When could this information have helped you 
or someone you know?” 
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Sample Instructional Assessment Task and Scoring Rubric 

EFF Read With Understanding Performance Level 3 



Determining Minimum-Wage Rights 





Task Description 




Step 5, continued 


Document evidence of linking new information 
to prior experience, if relevant. (Note that this 
activity may not be applicable to all learners, so 
while they may benefit from the opportunity, it 
can’t be used fairly to compare one learner’s 
performance to another’s.) 

Part 2 of the Task 




Step 6 


Ask the learner to read a scenario 
describing an employee who has recently 
started a job. Then ask the learner to write 
down the problem the main character (Marla) 
faces In her new job and what she can do to 
get assistance, according to the poster: 

“Now we are going to check how well you 
can connect the information on the poster to 
a real-life situation. Please read this story to 
yourself. When you finish reading, write down 
on your Answer Sheet in your own words the 
problem the main character (Maria) has in her 
new job. Then look back at the poster. In the 
information on the poster, find and write down 
on your Answer Sheet one resource she can go 
to for assistance.” 




Step 7 


Ask the learner to read the scenario aloud 


In an instructional 




and then discuss: 


context, this last part 




• The problem 


might be done as a 




• The resource the learner Identified from the 


role-playing activity. 




poster 


The teacher or an adult 




• One suggestion the learner would give the 


learner (maybe reading 




woman in the scenario 


the scenario) might take 
the role of Maria and, in 




“Please read the scenario to me out loud. When 


response, another leaner 




you are finished, please tell or read to me the 


would point to and say 




problem you wrote down and the resource you 


aloud the contact 




found In the information from the poster.” 


information from the 
poster that would help 




“Now, based on what you have read on the 
poster and in this scenario, please tell me what 
you can tell Marla about how to get help with 
her problem.” Document observations. 


Maria. 
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Sample Instructional Assessment Task and Scoring Rubric 

EFF Read With Understanding Performance Level 3 



Determining Minimum-Wage Rights 



Task Description 



step 8 



Debrief the performance of the task 
together. Share the scoring rubric and learner 
scores to give feedback on how well the 
learner(s) used the Read With Understanding 
Standard to perform the task, what knowledge 
and strategies they used well, and what they 
struggled with. Learners also use the scoring 
rubric to help them reflect on their own 
performance. Together, plan for next steps in 
instruction based on the performance. 



This is an instance 
where instructional 
assessment allows for 
more Immediate results 
— and more kinds of 
Information — to be 
available to teachers and 
learners than Is possible 
with accountability 
assessments. 
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Scoring Rubric for the Instructional Assessment Task: 
“Determining Minimum-Wage Rights” 

Read With Understanding, Performance Level 3 



Unable 

to Score Beginning Proficient Advanced 





• Decodes and recognizes 
some everyday and some 
specialized work-related 
words; needs to use 
dictionary often 


• Decodes and recognizes most 
everyday work-related words and 
phrases (such as “minimum 
wage,” “employer,” “employee”) 
and some specialized, multi- 
syllabic vocabulary (such 
as “compensation”); may 
sometimes use a dictionary 


• Easily decodes and 
recognizes specialized 
words without using a 
dictionary 




• Strategies for monitoring 
and enhancing 
comprehension need 
improvement; oniy 
restates one or two 
important points from 
poster text; may not 
correctiy answer 
questions based on 
text 


• Effectively monitors and enhances 
comprehension by: 

— recalling and restating at least 
three out of four important 
points from the poster text 

— locating and explaining the 
information in the poster and 
scenario text to correctly 
answer two brief questions 
and offer one suggestion 


• Shows superior monitoring 
and comprehension skills; 
accurately restates four 
or more important points 
from text; answers 
questions; provides detail 
in response to questions 
asking for suggestions 
or explanations 




• Reads aloud slowly and 
hesitantly; may need more 
than one attempt, but 
completes task with some 
direction and support 


• Reads aloud accurately and 
easily; completes task with little 
direction and support 


• Reads aloud without 
hesitation; completes task 
without direction or support 
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Scenario for “Determining Minimum-Wage Rights” 

Maria is 42 years old. She just started a new job two 
weeks ago. She is being paid $4.50 an hour for her 
work. Maria’s boss told her that if things go well and 
she works hard, he would move her up to minimum 
wage after six weeks. Maria feels like this isn’t fair but 
she’s willing to work hard and she needs this job. 



OR 



Role-Playing Script for “Determining Minimum-Wage 
Rights” 

“I’m Maria and I’m 42 years old. I just started a new job 
two weeks ago. I’m getting paid $4.50 an hour. 

“My boss says that if things go well and I work hard, 
he’ll move me up to minimum wage after six weeks. 

“Well, I do work hard and I don’t think it’s fair to make 
me wait that long. But I sure need this job so I guess 
there’s nothing I can do.” 



148 



EFF ASSESSMENT CONSORTIUM 



Improving Performance, Reporting Results 



Section 5: Guide To Using EFF Assessments To Improve Teaching and Learning 



Assessor Observation Form 

“Determining Minimum-Wage Rights” 



LEARNER NAME DATE. 



Learner Performance 


Notes on Performance 

(Strategy Use, Fluency, Independence) 


Reads the top section of the 
poster out loud 




Reads or says the two 
important points he or she 
wrote down about the top 
section of poster and/or 
offers further explanation 
with prompting 




Relates information from 
reading to prior experience 
(if relevant) 




Reads the scenario out loud 




States what he or she can 
tell Maria about how to get 
help with her problem 
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Answer Sheet 

“Determining Minimum-Wage Rights” 



LEARNER NAME DATE 

Part 1 

After reading the top section of the poster, write in your own words: 

• Two important points made on the poster about wages 
1 . 

2 . 

• Two important points made on the poster about enforcement of the law 
1 . 

2 . 



Part 2 

After reading Maria’s story, write in your own words the problem Maria has in her new job. 



Look back at the poster. Find and write down one resource Maria can go to for assistance. 
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Your Rights Under the Fair Labor Standards Act 



Federal Minimum Wage 




beginning October 1, 1996 beginning September 1, 1997 



Employees under 20 years of age may be paid $4.25 per hour during their first 90 consecutive calendar 
days of employment with an employer. 

Certain full-time students, student learners, apprentices, and workers with disabilities may be paid 
less than the minimum wage under special certificates issued by the Department of Labor. 

Tip Credit - Employers of “tipped employees” must pay a cash wage of at least $2.13 per hour if they 
claim a tip credit against their minimum wage obligation. If an employee’s tips combined with the 
employer’s cash wage of at least $2.13 per hour do not equal the minimum hourly wage, the employer 
must make up the difference. Certain other conditions must also be met. 

Overtime Pay 

At least 11/4 times your regular rate of pay for all hours worked over 40 in a workweek. 

Enforcement 

The Department of Labor may recover back wages either administratively or through court action, 
for the employees that have been underpaid in violation of the law. Violations may result in civil or 
criminal action. 

Lines of up to $10,000 per violation may be assessed against employers who violate the child labor 
provisions of the law and up to $1,000 per violation against employers who willfully or repeatedly 
violate the minimum wage or overtime pay provisions. This law prohibits discriminating against or 
discharging workers who file a complaint or participate in any proceedings under the Act. 

Note: • Certain occupations and establishments are exempt from the minimum wage and/or 
overtime pay provisions. 

• Special provisions apply to workers in American Samoa. 

• Where state law requires a higher minimum wage, the higher standard applies. 



For Additional Information, Contact the Wage and Hour Division office nearest you - listed in your 
telephone directory under United States Government, Labor Department. 



The law requires employers to display this poster where 
employees can readily see it. 


U.S. Department of Labor 




Employment Standards Administration 


WH Publication 1088 


Wage and Hour Division 


Revised October 1996 


Washington, D.C. 20210 
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Abilities — enduring attributes of individuals that 
influence skiU development and performance (e.g., 
cognitive, psychomotor, physical and sensory). 

(See Stein, 1997, p. 32) 

Accountability — in this context, accountability 
refers to ways in which adult education programs 
are answerable to all their customers, starting with 
the adult learners. Generally, programs are held 
responsible to their customers for reporting learner 
outcomes such as learning gains (using pre and 
postassessments), obtainment or retention of 
employment, placement in secondary education, 
and achievement of a GED or equivalent diploma. 
Accountability is a means to help measure a 
program’s quahty or effectiveness. 

Adult Learner — an adult who is engaged in formal 
education or training to build his or her knowledge 
and skills. This education or training includes the 
full range of education from basic education; literacy; 
and high school equivalency through postsecondary 
education and training, work-based training, 
proprietary school training, and certification 
programs. (See Stein, 1997, p. 29) 

Applied Learning Standards — refer to the EFF 
Gontent Standards as they are applied (or used) by 
adults in learning activities that involve a purposeful 
apphcation of knowledge, skills and strategies to 
accomplish meaningful tasks. 

Assessment — in educational settings, an assessment 
is a test or measurement used for one of three main 
purposes: to provide diagnostic information, to show 
learner progress or to evaluate a program. An EFF 
performance-based assessment would measure how 
well an adult can use an EFF Standard and its 
integrated skills to accomplish a meaningful goal. 

(See Mislevy & Knowles, eds., 2002, p. 36) 



Certified Rater — a person who participates in an 
EFF-sponsored training on how to administer and 
score accountability assessment tasks and is approved 
by the EFF staff after the training. 

Components of Performance — the term EFF uses 
to describe what a skill looks like when applied in the 
real world. The Gomponents of Performance for each 
skill offer a shared definition of what that skill entails. 
They reflect a problem-solving approach to skill 
development, enabhng learners and practitioners to 
focus on learning skills in such a way as to build the 
cognitive and metacognitive strategies required for 
lifelong learning and apphcation. The Gomponents 
of Performance are the “heart” of the EFF Standards 
and are designed to be the focus of planning, learning 
and assessment. (See Stein, 2000, p. 19-23) 

Construct — “the concept or the characteristic that a 
test is designed to measure” (AERA/APA/NCME, 1999, 
p. 173). Each EFF Standard has a construct — a set of 
targeted abilities — that defines and makes clear the 
underlying skill process for that Standard. 

Constructivism — a theory of learning that holds 
that all knowledge is constructed from previous 
knowledge. EFF embraces a school of constructivism, 
which invites learners to create their own meaning and 
achieve their own goals by interacting actively with 
objects and information and by linking new material 
to existing cognitive structures. Constructivism is 
based on the work of Jean Piaget, Jerome Bruner, Fev 
Vygotsky and John Dewey. (See Bingman & Stein, 
2001; Bransford, et. ah, 1999, p. 11; Merrifield, 1999, 
p. 9-11; University of Colorado at Denver School of 
Education Web site at http://carbon.cudenver.edu/- 
mryder/itc_data/constructivism.html 
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Content Standards — term used in a variety of fields 
to describe what individuals need to know and be able 
to do for a particular purpose. In EFF, the 16 Content 
Standards identify what adults need to know and be 
able to do to meet their purposes for learning and to 
be effective in their adult roles. Each EFF Content 
Standard comprises the title of the Standard and the 
Components of Performance for that Standard. 

(See EFF Standards. See Stites, 1999, p. 3; Stein, 2000, 
p. 19-20) 

Continuum of Performance — a description of what 
a particular skill looks like in a lifelong, developmental 
process of growing adult competence of performance. 
EFF is particularly interested in ensuring that the 
EFF continuum is sensitive to performances at the 
beginning of the continuum, but the aim is to portray 
for each standard a continuum that describes 
performances from novice to expert levels. The 
continuum is built around the four Dimensions of 
Performance, and performance levels are defined by 
identifying key features of performance at various 
points along the continuum. (See McGuire, 2000; 
Stein, 2000, p. 58-59) 

Dimensions of Performance — the theoretical 
foundation, based in cognitive science, on which 
the EFF Continuum of Performance for each skill is 
built. The Dimensions of Performance identify 
qualitative differences between novices and experts 
related to four areas: 1) Structure of knowledge base, 

2) Fluency of performance, 3) Independence of 
performance, and 4) Range of conditions for 
performance. (See Stein, 2000, p. 59-60; 

Bransford, et. al., 1999, Chapter 2) 

EFF Assessment Framework — a guide to an 
integrated process of measuring and reporting 
adult learners’ performance based on the EFF 
Content Standards. The Assessment Framework 
is in development and will comprise the Guiding 
Principles, the Dimensions of Performance, 
Performance Fevels for each Standard and 
Benchmark Tasks. (See Bingman & Stein, 2001) 



EFF Content Framework — all the elements 
identified through EFF research that can be used to 
support adults as they develop the skills, knowledge 
and abilities necessary to be effective in meeting their 
learning goals. The framework comprises the EFF 
Purposes, Role Maps, Common Activities, and 
Content Standards. (See Bingman & Stein, 2001) 

EFF Standards — the current EFF Standards are 
Content Standards (see Content Standards), which 
convey the knowledge and skills all adults need to be 
effective in meeting the four Purposes and in carrying 
out activities central to their roles as parents/family 
members, community members and workers. The 
16 Standards are based on the Generative Skills, which 
EFF research revealed were required to carry out the 
Common Activities and day-to-day tasks of adult 
living (see Generative Skills). Each EFF Standard 
comprises the skill name and the components of 
performance, which offer a shared definition of what 
the skill includes. The EFF Standards expand the 
field’s understanding of “hteracy” to include the 
Communication, Interpersonal, Decision-Making, 
and Fifelong Fearning skills. They are the heart of EFF 
and should be central to the planning, learning and 
assessment services of any adult learning program. 

(See Bingman & Stein, 2001; Stein, 2000, p. 15-25) 

EFF Teaching/Learning Cycle — a term that describes 
the teaching and learning process as four stages that 
teachers and learners move through in the learning 
environment: preparation, planning, carrying out the 
plan and reflection. The cycle consists of eight steps 
and a set of ongoing practices that are integrated 
throughout the process. For more information, 
see the EFF Teaching/Fearning Toolkit at 
http://cls.coe.utk.edu/efftlc/. This Web site includes 
tools and resources for using the EFF Teaching/ 
Fearning Cycle in adult education settings. 
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EFF Trainer Certification System — a performance- 
based trainer development and certification process 
that ensures that states and organizations 
implementing EFF have access to high-quality 
training and technical assistance services. For more 
information, contact Diane Gardner, Professional 
Development & Certification Coordinator for the 
EFF Center for Training and Technical Assistance, the 
Center for Fiteracy Studies, University of Tennessee, 
dgardner@utk.edu or 865-974-9949. (See Bingman & 
Stein, 2001) 

EFF Website — http://eff.cls.utk.edu 

Equipped for the Future (EFF) — a customer-driven, 
standards-based, collaborative initiative of the 
National Institute for Fiteracy to align the components 
of the nation’s adult learning system to focus on the 
range of skills and knowledge that adults need in their 
roles of family member, worker and community 
member to access information, express their own 
ideas and opinions, take independent action, keep up 
with a changing world, and exercise their rights and 
responsibifities. EFF has been instrumental in shifting 
approaches for adult literacy education from an 
emphasis on replicating K-12 education to one that 
uses research-based standards to prepare adults to 
meet their real-world goals. (See Bingman & Stein, 
2001; Merrifield, 1999, p. 1-4; Stein, 2000; Stein, 1997, 
p. 1, 30) 

Generative Skills — the term EFF gives to the skills 
or knowledge that research revealed to be core to the 
performance of a wide range of tasks carried out in 
multiple adult roles. The Generative Skills are durable 
over time in the face of changes in technology, work 
process and societal demands. They cross functions 
and serve as the foundation for effective adaptation to 
changes in role requirements. The Generative Skills 
became the basis for the EFF Content Standards. 

(See Stein, 2000, p. 15; Merrifield, 1999, p. 35; 

Stein, 1997, p. 30) 



High-Stakes Testing — high-stakes testing refers to 
assessments used to inform high-stakes decisions, such 
as reporting learner gains for education program 
accountability purposes. These tests are standardized 
to ensure reliability and validity. High-stakes testing 
differs from low-stakes testing, which occurs during 
instruction for the purposes of monitoring learner 
progress and identifying instructional needs (see 
Instructional Assessment). 

Instructional Assessment — a test used during 
instruction to monitor learner progress and identify 
instructional needs. The purposes for instructional 
assessments are formative (used to plan or inform 
learning and instruction) rather than summative 
(used to mark achievement at the end of instruction). 

Integrated Skill Process — also called the 
components of performance of an EFF standard, this 
term refers to the knowledge and skills incorporated 
into the definition for each EFF Standard. The 
integrated skill process describes what adults need 
to know and be able to do for a specific purpose. 

Inter-Rater Reliability — the extent to which 
independent scorings (by different raters) of an 
assessment are consistent with each other. 

Iterative — a term used in research to refer to the 
repetition of a cycle of processes with an eye toward 
moving ever more closely toward desired results. 

In EFF, the term is used to describe how EFF has 
progressively refined the concepts and components 
of EFF through research; feedback from customers 
(learners, practitioners, stakeholders and 
policymakers); incorporation of research 
developments in related areas; further feedback 
from customers; etc., in an effort to be responsive 
and credible to their constituents. (See Merrifield, 
1999, p. 4, 7-8) 

Knowledge — the foundation upon which skills and 
abilities are built. Knowledge refers to an organized 
body of information that can be applied to 
performance. Examples: mathematical concepts, 
languages, biology and finance. (See Stein, 1997, 

P-32) 
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Metacognition — refers to an individual’s ability to 
think about his or her own thinking and to monitor 
his or her own learning. Metacognition is integral to a 
learner’s ability to actively partner in his or her own 
learning and facihtates transfer of learning to other 
contexts. (See Bransford, et. ah, p. 12, 55-56) 

National Adult Literacy Survey (NALS) — a national 
survey reported in 1993 that provided a profile of the 
hteracy skills of the United States’ adult population. 
The results revealed that more than 40 percent of all 
American adults have literacy levels at Levels 1 or 2 
(out of 5), below the level required to secure jobs at 
good wages. 

National Institute for Literacy (NIFL) — an 

independent federal organization created by the 
National Literacy Act of 1991 to serve as a focal 
point for public and private activities that support 
the development of high-quahty regional, state and 
national literacy services. One of NlFL’s primary 
activities is promoting adult hteracy system 
reform through Equipped for the Future. 

(See http://www.nifl.gov) 

National Reporting System (NRS) — an outcome- 
based reporting system for the state-administered, 
federally funded adult education program required 
by Title II of the Workforce Investment Act of 1998. 
The goals of the NRS were to establish a national 
accountabihty system for education programs by 
identifying measures for national reporting and their 
definitions, establishing methods for data collection, 
developing software standards for reporting to the U.S. 
Department of Education, and developing training 
materials and activities on NRS requirements and 
procedures. EFF is working with the NRS to align 
outcomes required for federal reporting to the EFF 
standards. (See http://www.air-dc.org/nrs; Bingman 
& Stein, 2001; Stein, 2000, Chapter 5) 



Performance-Based Assessment — (also known as 
Performance Assessment) “product- and behavior- 
based measurements based on settings designed to 
emulate real-life contexts or conditions in which 
specific knowledge or skills are actually apphed” 
(AERA/APA/NCME, 1999, p. 179). Examples 
include problem-solving scenarios, journals, 
projects, performances, computer simulation tasks, 
portfolios and other approaches to measuring student 
learning that require the student to construct or 
produce a complex response. (See Ananda, 2000). 

Performance Indicators — the behavioral 
benchmarks (or indicators) that describe proficient 
performance on a standard at a particular 
performance level. The behavioral indicators show 
what proficient performance looks like along the four 
Dimensions of Performance. 

Performance Levels — a key threshold in adult 
learning and development that is characterized by 
observable changes in proficiency. These observable 
changes are noted by benchmarks (Performance 
Indicators). 

Performance Task — a carefully planned activity 
that requires learners to address all the components 
of performance of a standard in a way that is 
meaningful and authentic. Performance tasks can 
be used for both instructional and assessment 
purposes. (See Ananda, 2000; McGuire, 2000) 

Purposes for Learning — the four fundamental 
purposes that adults offer as reasons for furthering 
their literacy education. The Four Purposes for 
Eearning are 1) Access and Orientation, 2) Voice, 3) 
Independent Action, and 4) Bridge to the Future. EFF 
identified the purposes in 1994 through analysis of 
1,500 essays solicited from adult learners in response 
to the question, “What is it that adults need to know 
and be able to do in order to be literate, compete in 
the global economy, and exercise the rights and 
responsibilities of citizenship?” These purposes drive 
learning across the different contexts of adult life and 
capture the social and cultural significance of learners’ 
specific, individual goals (Merrifield, 1999, p. 13-17). 
(See Stein, 1995; Stein, 2000, p. 5-6) 
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Reliability — the extent to which independent test 
scores for a particular assessment are free from error 
and consistent with each other. 

Scoring Criteria — EFF performance-based 
assessments include rubrics that are used for scoring 
(grading/evaluating) the learner’s performance on 
that assessment task. The scoring criteria are the 
descriptions of learner performance (behavior) that 
are used for this evaluation. 

Scoring Rubric — a scoring rubric is a guide to 
evaluating (scoring) evidence of performance 
produced in response to a performance-based 
assessment task. The scoring rubric for an EFF 
performance -based assessment describes three 
levels of learner performance: beginning, 
proficient, and advanced. 

Skills — developed capacities to perform physical 
or mental tasks. (See Stein, 1997, p. 32) See also 
Generative Skills and EFF Standards. 

Standardized Tests — assessments designed for 
large-scale use. Conditions for administering and 
scoring standardized tests are established by the test 
publisher and must be strictly followed to produce 
consistent (rehable) and comparable results across 
multiple settings. These tests are often have a 
selected-response (such as multiple choice) format 
and may have automated scoring and reporting 
features. 



Technical Assistance — a service provided by the 
EFF National Center to support organizations in 
1) clarifying their goals and objectives for EFF 
implementation; 2) designing an implementation 
plan; and 3) providing training and on-going 
assistance related to curricular, instructional and 
administrative issues. (See Bingman & Stein, 2001) 

Transfer of Learning — the ability to extend what has 
been learned in one context to new contexts. Research 
has shown that when a subject is learned in multiple 
contexts, with opportunities to abstract general 
principles, transfer to new situations is increased. 

(See Bransford, et. ah, 1999, Chapter 3) EFF 
encourages programs to create learning situations 
that invite learners to use skills in a variety of contexts 
for a variety of purposes. (See Stein, 2000, p. 20) 

Validity — “the degree to which accumulated 
evidence and theory support specific interpretations 
of test scores entailed by the proposed uses of the test” 
(AERA/APA/NCME, 1999, p. 184). In other words, the 
degree to which a test measures what it is intended to 
measure and the degree to which test results provide 
meaningful and relevant information on learner 
performance. 
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